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Preface 


The  report  of  investigation  (ROI)  is  central  to  the  clearance  process.  It  documents 
the  background  investigation  and  is  the  primary  material  for  determining  security 
eligibility.  Given  its  importance,  PERSEREC  has  carried  on  a  research  effort  that 
examines  the  ROI  as  the  medium  of  communication  between  investigators  and 
adjudicators. 

Our  major  focus  in  this  project  has  been  the  organization  of  case  material  and  how 
it  affects  adjudicators’  processing  of  information.  From  the  outset  we  have  presumed 
that  a  uniform  format  for  the  ROI  will  promote  consistency  in  adjudication. 

The  following  report  presents  the  results  of  an  experimental  test  of  different 
formats  of  ROIs  on  adjudicator  decisions.  This  report  is  the  second  part  of  a  two-volume 
piece.  The  first  volume  (Suchan,  1993)  analyzes  the  effects  of  ROI  format  on 
adjudicators’  decisions.  We  suggest  the  reports  be  read  together. 

The  reports  are  of  interest  on  a  number  of  levels.  At  a  primary  level  the  studies 
show  there  is  consistency  in  the  final  adjudicative  decisions  reached  by  different  agencies 
in  the  Department  of  Defense  regardless  of  the  writing  style  of  the  ROI. 

At  a  secondary  level  the  studies  show  that  adjudicative  organizations  reach  these 
consistent  decisions  through  varied  and  distinctive  processes.  Each  facility  has  a 
recognizable  way  of  cognitively  processing  the  ROI.  Furthermore,  these  studies  show 
that  the  ROI  is  the  vehicle  that  connects  the  investigation  and  adjudication  communities 
and  the  written  style  of  the  ROI  influences  the  attitudes  and  opinions  of  adjudicators.  In 
this  sense  the  ROI  conveys  the  competence  and  skill  of  the  investigator. 

The  exciting  part  of  this  research  for  PERSEREC  is  that  these  studies  have  used 
the  ROI  as  a  diagnostic  probe  and  have  afforded  a  view  of  the  internal  workings  of  the 
adjudication  process.  We  hope  you  find  them  stimulating. 


Roger  P.  Denk 
Director 
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Executive  Summary 


This  project  examined  the  impact  of  document  design,  organization,  and  writing  style 
of  a  Report  of  Investigation  (ROI)  on  the  adjudicators  charged  with  assessing  them. 
Specifically,  the  adjudicators’  assessment  time,  impression  of  case  information,  perception 
of  the  field  agent  writing  the  cases,  and  writing  style  preferences  were  analyzed.  A  total 
of  40  Army,  Navy,  DISCR,  and  Air  Force  adjudicators  were  tested  using  both  "high-"  and 
"low-impact"  style  ROIs.  A  four-part  questionnaire  captured  their  assessment  times, 
impressions,  perceptions,  and  preferences  between  the  two  different  style  cases. 

The  results  indicate  adjudicators  prefer  the  high-impact  style  because  they  believe 
that  it 

*  presents  information  in  a  more  readable  format 

*  makes  it  easier  to  read,  remember,  and  apply  adjudication  criteria  to  case 
information,  particularly  information  contained  in  subject  and  developed 
reference  narrative  sections 

*  decreases  the  amount  of  rereading  needed  to  understand  case  information 

*  causes  less  fatique  while  reading 

Furthermore,  adjudicators  perceived  field  agents  who  wrote  high-impact  ROIs 
differently  from  those  who  wrote  low-impact  reports.  Specifically,  they  believe  high- 
impact  report  writers  possess  better  judgment,  are  better  organized,  and  are  more 
precise,  careful,  clear  thinking,  and  efficient  than  low-impact  writers. 

Although  adjudicators  believe  the  high-impact  style  makes  their  information 
processing  tasks  more  efficient  and  they  have  a  more  positive  impression  of  the  high- 
impact  report  writer  than  the  low-impact  one,  there  is  no  statistical  evidence  to  show  that 
the  high-impact  ROIs  decreased  adjudication  time.  However,  small  sample  size  and 
limitations  of  the  study’s  research  design  require  that  adjudication  time  results  be 
assessed  carefully. 


Policy  Implications 

Many  complex  factors  affect  the  timely  and  accurate  adjudication  of  cases.  Perhaps 
the  most  crucial  factor  is  the  needs  of  adjudicators.  This  research  has  shown  that 
adjudicators’  information  processing  needs  can  be  better  met  by  ROIs  that  are  organized 
and  written  in  the  high-impact  style.  This  adjudicator  need  has  largely  been  overlooked 
because  of  lack  of  awareness  about  different  strategies  for  organizing  and  presenting  ROI 
information. 
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What  specific  outcomes  will  result  from  meeting  adjudicators’  information  processing 
needs  still  remains  somewhat  fuzzy.  Although  the  high-impact  style’s  effect  on 
adjudication  time  is  unclear,  this  research  does  show  that  adjudicators  perceive  field 
agents  who  write  high-impact  ROIs  as  more  credible  than  those  who  use  current  ROI 
style;  this  difference  in  credibility  may  affect  adjudicators’  requests  for  additional 
information.  Furthermore,  the  high-impact  style  causes  less  reading  fatigue,  which  may 
affect  job  satisfaction  and  motivation.  Clearly,  this  relationship  between  writing  style  and 
outcomes  needs  careful  examination  before  DIS  establishes  ROI  writing  style  and 
organization  of  information  requirements  and  task  DODSI  to  provide  field  agents 
training  in  this  new  style. 

What  is  imperative  is  that  this  research  be  conducted  well  before  DIS  transitions  to 
an  automated,  computer-mediated  environment  where  adjudicators  will  be  reading  ROIs 
on  CRT  screens.  Research  on  electronic  mail  has  indicated  a  strong  connection  between 
the  oragnization  and  format  of  information  on  the  CRT  screen  and  readers’  ability  to 
process  that  information  and  their  perception  of  its  quality.  High-impact  E-Mail  messages 
are  perceived  more  positively  than  their  low-impact  counterparts.  In  short,  the  transition 
to  a  CRT  reading  environment  may  require  field  agents  to  develop  high-impact  writing 
skills. 


Recommendations 

This  study’s  results  suggest  the  following  additional  research: 

1.  Conduct  a  study  focused  on  adjudicator  reading  time  when  processing  high  and 
low-impact  derogatory  sections  of  ROIs.  To  conduct  this  study,  the  reasearch 
design  of  this  study  would  need  to  be  modified  to  isolate  the  adjudication  time 
variable,  the  sample  size  increased,  and  perhaps  the  cases  changed. 

2.  Replicate  this  study,  with  minor  modifications  in  research  design,  using  a 
CRT  screen  rather  than  paper  as  the  communication  medium.  Also,  increase 
sample  size  to  improve  the  predictive  power  of  the  results. 
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Introduction 


In  recent  years,  the  Department  of  Defense  (DoD)  has  sought  to  improve  the 
personnel  security  screening  process.  This  desire  was  prompted  primarily  by  the  rash  of 
security  breaches  over  the  last  several  years.  One  area  where  improvement  is  desired  is 
information  assessment.  This  assessment  occurs  when  personnel  at  adjudication  facilities 
evaluate  Personnel  Security  Investigation  (PSI)  documentation  obtained  from  the 
Defense  Investigative  Services  (DIS). 

The  Report  of  Investigation  (ROI)  is  the  primary  information  conduit  between 
these  agencies.  More  specifically,  the  ROI  is  the  written  interface  between  the  DIS  field 
agent  who  conducted  the  PSI  and  the  adjudicator  responsible  for  assessing  the  case. 

One  objective  of  all  involved  in  the  security  screening  process  is  to  improve  the 
quality  and  clarity  of  the  ROI.  ROI  improvements  will  hopefully  increase  reader 
comprehension  and  general  document  effectiveness.  Several  important  factors  that  affect 
the  quality  of  the  ROI  are  the  document  design,  organization,  and  writing  style. 

Two  studies  (Suchan  1989a,  1993)  have  examined  these  factors  for  the  Defense 
Personnel  Security  Research  Center  (PERSEREC).  Suchan’s  research  is  outlined  in  the 
1993  PERSEREC  technical  report  entitled  "Comparison  of  High-  and  Low-impact 
Report  of  Investigations  on  Adjudicators’  Decision  Making:  A  Pilot  Study."  His  primary 
focus  in  that  study  is  to  determine  the  effect  that  organization,  document  design,  and 
stylistic  factors  have  on  the  decision-making  ability  of  adjudicators.  This  report  builds  on 
that  work.  It  specifically  examines  how  the  ROI  document  design,  organization  and 
writing  style  affect  the  following: 

•  The  impressions  of  the  adjudicators  concerning  the  case  information 

•  The  perception  of  the  adjudicators  about  the  field  agent  who  wrote  the  case 

•  The  preference  of  the  adjudicators  for  one  of  the  two  case  styles  used  in  the  study 

•  The  case  processing  time  required  by  the  adjudicators. 
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Background 


As  stated  earlier,  this  report  continues  the  ongoing  research  of  James  Suchan 
(Suchan,  1993).  His  research  is  concerned  with  the  adjudicators’  ability  to  accurately 
read  ROIs  and  make  reliable  and  consistent  judgments  about  a  subject’s  security 
worthiness.  His  objective  is  to  determine  if  changes  in  document  design,  organization, 
and  writing  style  will  enable  adjudicators  to  make  higher  quality  decisions.  This  decision¬ 
making  ability  of  the  adjudicator  is  the  crux  of  the  personnel  security  process,  the 
purpose  of  which  is  to  weed  out  subjects  who  are  capable  of  committing  espionage. 

Drawing  from  the  existing  body  of  literature  that  defines  the  qualities  of  a  clearly 
written  document,  Suchan  composited  those  qualities  into  two  ROIs.  The  clearly  written 
ROIs  contained  organizational  features  such  as  internal  previews  and  clear  transitional 
tags;  document  design  characteristics  such  as  strategic  use  of  lists,  bullets,  headings  and 
white-space;  and  stylistic  traits  such  as  active  verbs  and  right-branching  modifiers.  These 
features  are  commonly  referred  to  as  a  high-impact  style. 

The  converse  of  the  high-impact  style  is  the  low-impact  style.  This  is  the  style  that 
field  agents  currently  use  when  writing  ROIs.  This  style  includes  features  such  as  poor 
document  design,  "wall-to-wall"  prose,  no  headings  or  lists,  and  no  use  of  bold  print  or 
other  highlighting  devices;  lack  of  internal  previews  at  the  beginning  of  long  sections; 

Jong  convoluted  sentences  that  strain  short-term  memory;  passive  verbs;  and  finally,  lack 
of  old  information/new  information  patterns  to  create  coherence. 

Suchan  tested  both  the  high-impact  and  low-impact  ROIs  on  a  sample  of  40 
adjudicators  from  four  adjudication  sites:  12  from  the  Army  Central  Adjudication  Facility, 
12  the  Navy  Central  Adjudication  Facility,  4  from  the  Air  Force  Security  Clearance 
Office,  and  12  from  the  Directorate  for  Industrial  Security  Clearance  Review  at 
Columbus,  Ohio.  His  objective  was  to  determine  whether  differences  in  ROI  design, 
organization,  and  writing  style  affected  the  decision-making  ability  of  the  tested 
adjudicators. 

A  four-part  questionnaire  was  used  to  collect  the  necessary  data  for  the  research, 
although  Suchan  only  analyzed  Part  I  of  that  questionnaire.  The  remaining  three  parts  of 
the  questionnaire  collected  data  concerning  the  following: 

•  The  impressions  of  the  adjudicators  concerning  the  case  information  (Park  II) 

•  The  perception  of  the  adjudicators  about  the  field  agent  who  wrote  the  case  (Part 

III) 

•  The  preference  of  the  adjudicators  for  one  of  the  two  case  styles  used  in  the  study 

(Part  IV) 
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Suchan’s  research  results  provide  the  basis  for  this  report.  The  unexamined  data 
from  the  remaining  three  parts  of  the  questionnaire  were  used  as  the  primary  source  of 
data  for  the  analysis.  The  results  of  Suchan’s  study  are  outlined  in  the  following 
Literature  Review,  while  a  more  detailed  description  of  his  research  design  is  included  in 
Research  Methodology. 


Research  Questions 

This  report  will  answer  the  following  primary  and  secondary  research  questions. 

Primary’  Research  Questions.  The  objective  of  this  report  is  to  analyze  the 
remaining  Parts  II,  III,  and  IV  of  the  Suchan  questionnaire.  Specifically,  the  report  will 
determine  if  there  is  a  statistical  significance  in  the  questionnaire  responses  for  ROIs 
written  in  a  high-impact  style  versus  low-impact  style.  The  primary  research  questions 
are  as  follows: 

•  Will  there  be  statistically  significant  differences  between  adjudicators’  impressions 
of  their  ability  to  process  case  information  presented  in  a  high-impact  ROI  versus  a 
low-impact  one? 

•  Will  there  be  statistically  significant  differences  between  adjudicators’  perceptions 
or  feelings  towards  the  field  agents  who  wrote  a  high-impact  ROI  as  compared  to 
those  who  wrote  the  low-impact  ROI? 

•  Will  there  be  statistically  significant  differences  in  adjudicators’  preferences  for  the 
case  information  presented  in  a  high-impact  versus  the  low-impact  style? 

•  Will  there  be  statistically  significant  differences  in  the  adjudicators'  processing  time 
of  the  case  information  presented  in  a  high-impact  style  versus  the  low-impact 
style? 


Secondary  Research  Questions.  The  report  will  answered  the  following  secondary 
research  questions: 

•  Will  there  be  any  statistical  differences  in  the  primary  research  questions  presented 
above,  when  results  from  the  adjudicators  at  the  various  central  adjudication  sites 
are  analyzed  independently? 

•  Will  there  be  any  statistical  differences  in  the  primary  research  questions  presented 
above,  when  the  population  of  adjudicators  is  broken  down  demographically? 
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Chapter  Preview 


This  section  discusses  the  organization  of  the  remainder  of  the  report  and  previews 
each  coming  chapter. 


Literature  Review.  An  outline  of  the  results  of  prior  related  research  and  a 
discussion  of  the  significant  published  literature  on  the  subject  of  document  design, 
organization,  and  writing  style  will  be  presented. 

Research  Methodology.  A  detailed  description  of  the  research  methodology  is 
presented.  Both  case  selection  and  questionnaire  content  are  examples  of  topics 
discussed  in  this  chapter. 

Presentation  and  Analysis  of  Adjudication  Times  and  the  Adjudicator’s  Impressions 
of  Case  Information.  A  presentation  and  analysis  of  the  adjudication  times  and  Part  II  of 
the  questionnaire  is  the  focus  of  the  chapter.  Part  II  covers  the  impressions  of  the 
adjudicators  concerning  the  case  information. 

Presentation  and  Analysis  of  Questionnaire  Part  III  Results:  the  Adjudicator’s 
Perceptions  of  Field  Agents.  Similar  to  the  preceding  chapter,  an  analysis  of  Part  III  of 
the  questionnaire  and  discussion  of  the  statistical  testing  results  is  presented.  Part  III 
tests  the  perception  of  the  adjudicators  about  the  field  agents  who  wrote  the  case. 

Presentation  and  Analysis  of  Questionnaire  Part  IV  Results:  the  Writing  Style 
Preferences  of  Adjudicators.  This  chapter  present  the  result  of  Part  III  of  the 
Questionnaire.  Part  III  determine  the  adjudicators  preferences  between  the  high  and 
low-impact  writing  style. 

Conclusions  and  Recommendations.  This  chapter  summarizes  the  conclusions  and 
recommendations  of  the  report. 
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Literature  Review 


Chapter  Introduction 

This  chapter  focuses  on  the  significant  published  literature  concerning  the  subject 
of  document  design,  organization,  and  writing  style.  Many  of  the  document  improvement 
strategies  discussed  in  this  literature  were  incorporated  into  the  revised  high-impact 
ROIs.  The  literature  review  will  be  presented  in  three  sections.  The  first  section 
outlines  the  published  body  of  literature  associated  with  the  document  factor  stated 
above.  The  second  section  shows  the  limited  amount  of  literature  available  in  the 
following  areas: 

•  Reader  impressions  of  written  communication 

•  Reader  perception  of  the  writer  of  a  written  communication 

•  Reader’s  writing  style  preferences 

Finally,  the  third  section  gives  a  detailed  description  of  the  results  of  several 
PERSEREC  studies  in  the  area  of  ROI  document  design,  organization  and  writing  style. 
Included  in  this  third  section  will  be  a  discussion  of  the  results  of  the  Suchan  study  of 
which  this  report  is  a  continuation. 


Document  Factors 


Readability  Formulas 

Readability  formulas  are  important  because  for  decades  researchers  have  used 
them  as  a  dominant  tool  for  measuring  document  effectiveness.  This  study  avoided  using 
these  formulas  to  determine  the  relative  readability  and  efficiency  of  current  ROIs 
because  of  their  significant  limitations.  One  particular  researcher  (Seltzer,  1981)  best 
summarizes  their  limitations. 

Seltzer  argues  that  because  readability  formulas  only  assess  sentence  length  and 
word  choice  they  fail  to  account  for  important  reader  comprehension  factors.  These 
factors  include: 

•  the  reader’s  interest  in  the  document 

•  the  reader’s  familiarity  with  a  specialized  language  that  might  be  used  in  a 
document 

•  the  effect  that  convoluted  sentences,  poorly  placed  modifiers,  and  other  syntactic 
difficulties  have  on  a  reader’s  ability  to  process  a  sentence 
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•  the  reader’s  ability  to  see  easily  logical  association  between  groups  of  sentences 

and  paragraphs 

Another  important  research  study  (Hirsch,  1977)  points  out  other  flaws  in  these 
formulas.  Hirsch  argues  that  readability  formulas  fail  to  consider  several  important 
factors  that  contribute  to  a  document’s  effectiveness.  He  believes  that  to  make  a 
document  readable  and  easy  to  understand  the  writer  must  use  organizational,  syntactic, 
and  linguistic  strategies  that  will  constrain  meaning.  This  is  accomplished  by  reducing  the 
semantic-syntactic  possibilities  that  can  occur  in  a  clause,  sentence,  paragraph,  or  even  a 
document.  Hirsch  also  contends  that  subject-verb  patterns,  the  voice  of  verbs,  clause 
length,  and  transitional  tags  can  be  used  in  conjunction  with  each  other  to  improve 
document-processing  speed  and  comprehension. 

It  is  clear  that  there  are  severe  limitations  to  using  readability  formulas  when 
assessing  the  effectiveness  of  a  document.  Seltzer  and  Hirsch  point  out  the  formulas’ 
flaws  and  their  argument  against  these  formulas.  However,  the  researchers  only  allude  to 
the  specific  factors  that  make  a  document  effective. 

Those  specific  factors  include  subject-verb-object;  transitional  tags  and  internal 
previews;  active  verbs;  shortened  paragraphs,  sentence  and  clause  length;  and  several 
others.  These  factors  were  used  to  generate  the  revised  high-impact  document  in  the 
Suchan  study. 

The  remainder  of  this  section  will  be  devoted  to  outlining  the  literature  that 
describes  the  above  stated  document  effectiveness  factors. 


Document  Organization 

Several  business  communication  researchers  have  made  significant  discoveries 
about  the  factors  that  influence  information  processing.  Two  particular  researchers 
(Fielden  &  Dulek,  1984)  have  studied  and  assessed  document  effectiveness  from  a 
macro-organizational  perspective.  They  argue  that  a  "bottom  line"  communication 
technique  decreases  reading  time  and  improves  document  comprehension.  The  essence 
of  this  technique  is  for  the  writer  of  nonsensitive  documents  to  state  in  the  first 
paragraph  the  purpose  of  the  document. 

Fielden  and  Intek  further  argue  that  to  decrease  reading  time  and  improve 
comprehension  in  longer  documents  the  writer  should  also  include  a  contract  sentence 
immediately  following  the  bottom  line.  The  contract  sentence,  basically  an  internal 
preview,  organizes  the  remainder  of  the  communication  for  the  reader  by  providing  a 
skeletal  framework  of  the  major  point  to  be  covered  and  the  order  in  which  they  will  be 
presented. 

Although  Fielden  and  Dulek  cite  no  empirical  evidence  to  support  their  claims 
about  the  advantages  of  bottom  lines  and  contract  sentences,  the  work  of  two  cognitive 
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psychologists  (Bransford  &  Johnson,  1972,  1973)  justify  the  value  of  these  two 
organizational  strategies.  Bransford  and  Johnson  found  that  subjects  who  were  provided 
a  concrete  title  to  a  very  general,  context-free,  written  passage  could  better  understand 
and  remember  the  passage  than  those  not  provided  a  title.  The  title  provided  a  mental 
model  that  helped  readers  classify  and  integrate  information.  Similarly,  a  bottom  line 
and  contract  sentence  may  help  a  reader  create  a  mental  model  of  a  document’s  content 
and  constrain  its  meaning. 

Another  study  supports  Fielden  and  Dulek’s  findings.  This  study  (Suchan,  1989a) 
assessed  the  communication  efficiency  of  the  high-impact  style  versus  the  traditional 
bureaucratic  low-impact  style  favored  by  the  government.  The  study  involved  262  naval 
officers.  Suchan  found  that  the  high-impact  style  took  the  officers  between  17%  and 
23%  less  time  to  read  than  the  bureaucratic  style,  and  produced  much  better  message 
comprehension. 


Informational  Design 

Another  document  factor  that  determines  effectiveness  is  visual  design.  Visual 
design  includes  the  writer’s  use  of  headings,  list,  bold  print,  underscoring,  spacing,  and  so 
on.  Several  researchers  (Kostelnick,  1988;  Hartley  &  Trueman,  1985;  Battison  & 
Landesman,  1981)  have  confirmed  that  using  these  strategies  improves  effectiveness. 

Kostelnick  found  that  local  design  strategies  (lists,  bullets,  indentation,  and  so  on) 
may  restrain  meaning  by  enabling  readers  to  code  and  chunk  information  more  easily. 
Hartley  and  Trueman  have  proved  through  extensive  testing  that  headings  improve 
reader  performance.  Battison  and  Landesman  have  shown  that  FCC  regulations  revised 
into  a  more  visually  appealing  question-answer  format  improved  reader  comprehension. 


Sentence  Level  Research 


Three  researchers  (Miller,  1956,  1970;  Bever,  1972;  Clark  &  Clark,  1968)  have 
studied  how  readers  use  short-  and  long-term  memory  to  process  sentences.  In 
particular,  Clark  and  Clark  found  that  because  a  reader’s  short-term  memory  can  hold 
only  about  seven  chunks  of  information,  readers  confronted  with  too  much  modifying 
detail  (left  embedding  of  information)  may  have  difficulty  retaining  in  short-term  memory 
the  primary  subject-verb  unit  of  the  sentence. 

This  research  justifies  the  need  for  writers  to  create  word  sequences  that  are  easy 
for  readers  to  form  into  stable  groups.  A  writer  should  use  the  subject-verb-object  order 
throughout  the  document  to  create  a  stable,  predictable  pattern  that  places  little  demand 
on  the  reader’s  short-term  memory.  The  use  of  these  sequences  should  increase  reading 
speed  and  improve  comprehension. 
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Active-Passive  and  Word  Choice  Research 


Three  studies  (Olson  &  Filby,  1972;  Danks  &  Sorce,  1973;  Charrow  &  Charrow, 
1979)  have  shown  that  readers  process  active  verbs  faster  than  passive  ones.  This  is 
because  passive  verbs  invert  the  typical  subject-verb-object  order  syntactic  pattern. 

Finally,  research  (Rosch,  1973,  1975)  has  shown  that  we  are  able  to  process  quickly 
and  accurately  language  that  represents  basic  classifications  or  prototypes.  Rosch  found 
that  language  which  is  relatively  concrete  ( e.g .  "cow"  versus  "bovine")  can  trigger  other 
clarifying  words  (abstract  background  information)  stored  in  long-term  memory.  What 
constitutes  concrete  language,  however,  may  be  organizationally  unique. 

For  example,  adjudicators  are  accustomed  to  reading  specialized  terms  such  as 
"PSI"  and  "ROI".  These  terms  trigger  a  range  of  concrete  associations  in  the  adjudicator 
but  not  in  a  reader  unfamiliar  with  security  issues.  In  short,  the  consistent  use  of  context - 
based,  organizationally  specific  specialized  terms  does  not  degrade  comprehension. 

These  results  support  the  assertion  that  if  a  writer  does  not  use  concrete  language 
as  defined  by  the  organizational  context,  the  reader  may  not  see  recognizable  prototypes 
and  thus  must  attempt  to  construct  them  while  reading.  This  construction  process 
increases  reading  time,  strains  short-term  memory,  reduces  comprehension,  and  degrades 
document  effectiveness. 


Reader  Perception.  Impression.  Preference  Research 

An  exhaustive  on-line  computer  search  was  conducted  to  determine  the  extent  of 
prior  empirical  research  on  readers  perception  of  writers  based  on  writing  style  and 
document  design.  Only  three  studies  were  located. 

Brown  and  Hemdl  (1986)  found  that  upper-level  managers  in  a  large  engineering 
firm  perceived  reports  written  in  a  bureaucratic  style  to  be  more  competent  and  objective 
than  reports  using  plain  language  guidelines  that  the  company  had  recently  implemented. 
Rogers  and  Hildebrandt  (in  press)  discovered  that  automotive  managers  strongly  resisted 
using  company  plain  language  criteria  in  their  dealer  contact  reports  because  they 
believed  they  would  be  perceived  by  their  bosses  as  being  less  thorough  and  analytical. 

Another  study  (McClure,  1990)  surveyed  the  lower  management  levels  of 
Sprint/United  Telephone  of  Florida  to  determine  if  these  managers  perceived  their 
reading  preferences  to  be  the  same  as  those  of  their  upper-level  managers.  McClure 
found  that  employees  were  more  aware  of  their  boss’s  reading  preferences  than  their 
CEOs.  Also,  as  communication  flowed  to  higher  levels  of  management,  the  employee 
wrote  longer  sentences,  used  bigger  words,  and  relied  on  the  passive  voice  in  order  to 
appear  more  formal,  respectful,  and  educated.  These  choices  led  to  a  increased  reading 
difficulty. 
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PERSEREC  Studies 


As  shown  in  the  earlier  sections  of  this  chapter,  there  exists  a  significant  body  of 
literature  that  defines  the  qualities  of  a  clearly  written  document.  PERSEREC  has 
conducted  three  studies  (Haag,  Schroyer  &  Crawford,  1989;  Suchan,  1989a,  1993)  that 
relate  to  adjudicators’  ability  to  process  and  assess  information. 


Haas.  Schrover.  and  Crawford  Study 

Haag,  Schroyer,  and  Crawford  found  that  adjudicators  were  not  satisfied  with  the 
way  information  is  presented  and  that  they  desired  the  reports  be  written  more  clearly. 
They  also  concluded  that  adjudicators  had  difficulty  locating  and  extracting  information 
from  narrative  segments  of  ROIs.  The  adjudicators  hoped  that  the  ROI  coul '  be 
formatted  to  parallel  and  support  the  adjudication  decision-making  process. 


Suchan  (1989a)  Study 

Similarly,  Suchan  also  found  that  the  adjudicators  were  dissatisfied  with  the  writing 
style  of  the  ROI.  He  used  talk-aloud  protocols  and  field  interviews  to  discover  that 
adjudicators  had  difficulty  sorting  and  classifying  information  in  the  narrative  sections  of 
ROIs  with  significant  derogatory  information.  Adjudicators  often  had  to  reread  the 
report  sections,  take  extensive  notes,  and  use  other  strategies  to  create  the  information 
patterns  necessary  for  them  to  apply  the  adjudication  criteria.  This  study  was  the 
precursor  to  the  1993  Suchan  study  on  which  this  report  is  based. 


Suchan  (19931  Study 

Suchan’s  (1993)  primary  objective  was  to  determine  whether  adjudicators  assess 
ROIs  written  in  a  high-impact  style  differently  from  the  same  ROIs  written  in  the  current 
low-impact  style.  Secondly,  Suchan  examined  whether  adjudicators  at  various  central 
adjudication  sites  assessed  differently  the  high-  and  low-impact  ROIs.  Furthermore,  he 
determined  if  adjudication  site  affected  the  degree  to  which  adjudicators’  decisions 
conformed  to  those  of  their  superiors. 

Suchan’s  results  contradict  previous  laboratory  research  that  show  clear  writing,  as 
exemplified  by  the  high-impact  style,  helps  readers  to  more  effectively  process 
documents.  The  data  show  there  are  no  statistically  significant  differences  in  decision 
type  between  the  high-  and  low-impact  ROIs.  Also,  high-impact  ROIs  did  not  result  in 
decisions  that  more  closely  matched  those  of  supervisory  adjudicators,  except  at  the  Army 
adjudication  site.  In  short,  the  high-impact  ROIs  did  not  help  adjudicators  make  better 
quality  decisions. 
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Research  Methodology 


Chapter  Introduction 

This  chapter  gives  a  detailed  outline  of  the  research  methodology  developed  and 
used  to  gather  data  for  this  study.  It  also  describes  the  techniques  used  by  the  author  to 
analyze  the  *  a  naming  three  parts  of  the  questionnaire.  The  topics  included  in  this 
chapter  are  as  follows:  case  selection,  content  and  treatment;  description  of  the 
adjudicator  sample  population;  testing  procedures;  questionnaire  content;  statistical 
methods  for  analyzing  the  questionnaire  results;  organization  of  analysis;  and  the 
limitations  of  the  methodology  and  analysis. 


Case  Selection.  Content,  and  Treatment 
Case  Selection 


Earlier  PERSEREC  research  (Wiskoff  &  Fitz,  1991)  found  that  adjudicators  dealt 
with  an  average  of  1.7  issues  per  case  and  assessed  cases  with  multiple  issues  46.3%  of 
the  time.  Consequently,  cases  that  contained  two  issues  were  used  in  this  study. 

Furthermore,  research  shows  that  adjudicators  most  often  dealt  with  Financial, 
Drugs/Alcohol,  and  Emotionai/Mental  issues.  One  of  the  most  frequent  double  issue 
combinations  was  drug  and  alcohol  (Lewis,  Koucheravy,  &  Carney,  1989;  Wiskoff  &  Fitz, 
1991). 


With  this  in  mind,  multiple  issue  cases  containing  combinations  of  Financial, 
Drugs/Alcohol,  or  Emotional/Mental  derogatory  information  were  requested  from  the 
Personnel  Investigation  Center  (FIC).  After  careful  reading  of  56  micro-fiche  cases  that 
met  these  criteria,  two  cases  were  chosen  that  best  met  the  multiple  issue  and  issue 
frequency  criteria.  These  cases  alee  reflected  the  document  design,  organizational,  and 
stylistic  characteristics  typical  of  ROIs  containing  derogatory  information  (Suchan,  1989a). 


Case  Content 


The  two  cases  selected  are  called  by  the  pseudonyms,  Czamek  and  Rokitka.  A 
brief  overview  of  each  case  is  described  below. 

Czamek.  Czamek  is  a  24-year-old  male  applying  for  a  Top  Secret  clearance 
required  for  a  security  guard  position.  The  major  derogatory  information  deludes: 

•  falsification  of  his  PSQ 
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•  aberrant  behavior  (lying,  personal  threats,  and  insubordination)  resulting  in  an 
honorable  discharge  from  the  military  after  only  5  weeks  in  service 

•  personal  and  on-the-job  dishonesty 

•  minor  criminal  conduct  resulting  in  loss  of  employment 

•  minor  financial  matters 

Rokitka.  Rokitka  is  a  49-year-old  female  applying  for  a  Top  Secret  clearance 
necessary  for  her  to  work  as  a  computer  programmer.  The  major  derogatory  information 
in  her  case  includes: 

•  alcohol/drug  abuse:  use  of  marijuana,  cocaine,  and  barbiturates 

•  criminal  conduct:  use  of  cocaine  on  the  job 

•  falsification  of  PSQ 


Case  Treatment 


Suchan  revised  both  the  Czamek  and  Rokitka  cases  using  document  design, 
organizational,  and  stylistic  strategies  that  research  has  demonstrated  leads  to  improved 
comprehension  (Duffy,  1985;  Redish,  1989;  Seigel,  1978;  Selzer,  1983;  Suchan,  1989b). 
Furthermore,  a  thorough  literature  review  was  conducted  tc  ensure  that  each  of  the 
independent  variables  used  in  revising  the  low-impact  ROIs  improved  comprehension. 
These  independent  vaiiables  include: 

•  Paragraph  Length:  paragraphs  were  no  longer  than  3-4  sentences;  one  sentence 
paragraphs  were  used  to  emphasize  important  information  (Felker,  Redish,  & 
Peterson,  1985). 

•  Headings  and  Subheadings:  major  headings  and  subheadings  were  used  to 
telegraph  major  sections  of  derogatory  information  (Reiish,  1989). 

•  Lists  and  Bullets:  these  graphic  aids  highlighted  significant  information,  broke  out 
statistical  information,  and  laid  out  complicated  chronological  sequences  (Benson, 
1985;  Rubens,  1986). 

•  Internal  Previews:  at  the  beginning  of  long  derogatory  sections,  an  internal  preview 
was  used  to  provide  a  quick  overview  of  the  major  points  that  were  to  come  in  that 
section.  The  language  in  the  preview  mirrored  the  language  in  the  major  headings 
(Redish,  Battison,  &  Gold,  1985). 
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•  Clauses  in  Subject-Verb-Object  Order:  subjects,  verbs,  and  objects  within  clauses 
were  kept  as  close  to  each  other  as  possible  to  avoid  strain  on  short-term  memory 
(Fodor  &  Garrett,  1967). 

•  Modifier  Strings  Right-Branched:  long  strings  of  modifying  information  were  placed 
to  the  right  of  the  object  (right-branching)  versus  before  the  subject  (left¬ 
branching).  Research  has  shown  that  right-branching  of  information  enables 
readers  to  read  faster  and  remember  more  easily  information  (Haviland  &  Clark, 
1974;  Miller  &  Isard,  1964). 

•  Active  Verbs:  when  appropriate,  passive  verbs  were  changed  to  active  verbs  to 
make  clear  the  agent  of  a  particular  action  (Mirel,  1988;  Selzer,  1983). 

•  Explicit  Transitional  Tags:  clear  transitional  tags,  often  left  out  in  narrative  report 
writing,  were  added  to  make  it  easier  for  readers  to  quickly  grasp  relationships 
between  ideas  and  sections  of  reports  (Guillemette,  1987). 

Only  material  that  field  agents  had  written  was  revised.  Information  contained  in 
subjects’  and  references’  statements,  medical  assessments,  and  military  reports  remained 
as  originally  written.  Also,  two  experienced  field  agents  reviewed  the  revised  cases  and 
noted  that  though  the  cases  "looked"  different,  there  were  no  differences  in  content. 

As  indicated  in  the  Introduction,  the  revised  cases  are  called  high-impact  and  the 
original  unmodified  cases  low-impact. 


Description  of  Sample  Population 

Forty  senior  adjudicators  (GS  11  through  GS  13)  from  four  major  central 
adjudication  sites  participated  in  the  study.  Adjudication  supervisors  at  each  site  chose 
the  adjudicators  to  be  tested.  Once  again,  the  breakdown  of  adjudicators  from  each  site 
is  listed  below; 

•  Army  Central  Adjudication  Facility  (12) 

•  Navy  Central  Adjudication  Facility  (12) 

•  Air  Force  Security  Clearance  Office  (AFSCO)  (4) 

•  Directorate  for  Industrial  Security  Clearance  Review  (DISCR)  at  Columbus  (12) 

A  high  caseload  backup  prevented  AFSCO  from  allowing  more  adjudicators  to 
participate  the  study. 
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A  total  of  31  female  and  9  male  adjudicators  read  the  cases.  Average  age  was  45, 
and  average  years  of  adjudication  experience  was  13.  The  typical  adjudicator  had  spent 
slightly  more  than  6  years  at  the  adjudication  site. 


Testing  Procedures 

The  adjudicators  were  tested  on  site.  At  each  site  they  were  divided  randomly  into 
two  groups,  with  each  group  adjudicating  two  cases. 

•  Group  I  (six  adjudicators  at  each  site  except  for  AFSCO)  received  a  revised,  high- 
impact  treatment  of  the  Czamek  case  and  an  original,  low-impact  treatment  of  the 
Rokitka  case. 

•  Group  II  received  a  low-impact  treatment  of  the  Czamek  case  and  a  high-impact 
treatment  of  the  Rokitka  case. 

A  counterbalanced  2X2  design  was  used  to  ensure  that  each  adjudicator 
responded  to  a  different  case  written  in  the  typical,  low-impact  style  and  the  revised  high- 
impact  style  and  that  order  of  presentation  would  not  influence  results. 

At  each  adjudication  site,  the  adjudicators  assessed  the  cases  in  a  large  room.  To 
guarantee  that  all  adjudicators  were  given  the  same  instructions,  the  researcher  read  a 
prepared  script  that  explained  the  purpose  of  the  exercise,  the  materials  they  were 
receiving,  and  process  they  were  to  use  in  assessing  the  cases  and  completing  the 
questionnaires. 


Content  of  Questionnaire 

Adjudicators  were  asked  to  complete  a  questionnaire  after  adjudicating  each  case. 

The  first  questionnaire  contained  three  parts: 

•  Part  I  contained  seven  open-ended  questions,  requesting  information  about  the 
adjudication  decision,  the  rationale  for  the  decision,  the  adjudication  criteria 
applied,  the  mitigating  criteria  applied,  the  need  for  more  information,  and  the 
perceived  need  to  send  a  Letter  of  Intent  (LOI). 

•  Part  II  contained  23  questions  about  the  readability  of  the  case,  the  ease  in  finding 
derogatory  information,  the  quality  of  field  agents’  writing,  and  so  on.  A  five-point 
bipolar  scale  was  used  to  capture  these  perceptions.  See  Appendix  A. 

•  Part  III  examined  adjudicators’  perceptions  or  feelings  toward  the  field  agents  who 
wrote  the  ROI.  Fifteen  semantic  differential  items  on  a  five-point  scale  were  used 
to  capture  this  data.  See  Appendix  B. 
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The  second  questionnaire,  completed  after  the  second  case  was  adjudicated,  was 
identical  to  the  first  except  it  contained  an  additional  fourth  part,  Part  IV,  that  asked 
adjudicators  which  case  was  easier  to  read,  required  less  rereading,  increased  confidence 
in  the  competence  of  field  agents,  and  so  on  (see  Appendix  C).  There  were  eight  case 
preference  questions  of  this  type  and  two  yes/no  questions. 

This  report  focuses  on  the  results  from  Part  II,  III,  and  IV  of  the  questionnaire. 


Statistical  Testing  of  Questionnaire  Results 

Software  Statistical  Package 

The  questionnaire  results  from  the  Suchan  study  were  recorded  for  analysis  in  a 
statistical  package  called  APL.  This  package  was  used  to  analyze  the  results  from  Part  I 
of  the  questionnaire.  The  analysis  of  the  remaining  parts  of  the  questionnaire  was 
completed  using  the  statistical  package  SPSS.  Much  care  was  taken  when  transferring 
the  questionnaire  results  from  APL  to  SPSS.  The  first  part  of  the  survey  was  re-analyzed 
using  SPSS  and  compared  with  the  previous  APL  results.  Since  the  result  were  the  same, 
the  integrity  of  the  data  was  maintained. 


Confidence  Interval 


Since  this  is  a  pilot  study  of  how  document  design,  organization,  and  writing  style 
impact  upon  the  impressions,  perceptions,  and  preferences  of  adjudicators,  a  5% 
confidence  interval  was  determined  as  significant  during  statistical  analysis  of  both  the 
high-  and  low-impact  Czamek  and  Rokitka  cases.  A  10%  confidence  interval  was 
deemed  sufficient  when  the  cases  were  analyzed  independently,  or  the  population  of 
adjudicators  was  stratified.  With  this  in  mind,  the  10%  confidence  interval  results  from 
our  analysis  need  to  be  interpreted  with  care. 


Statistical  Techniques 

ANOVAs.  When  analyzing  results  of  Part  II  and  III  of  the  questionnaire, 
ANOVAs  were  used  for  determining  the  statistical  significance  of  the  comparative  data 
from  the  high-impact  and  low-impact  reports.  The  dependent  variables  were  the  specific 
factors  coded  into  the  questions  themselves,  while  the  independent  variable  in  all  tests 
was  the  specific  report  writing  style  ( Le .,  high-impact  or  low-impact).  An  example  of  a 
dependent  variable  from  Part  II  would  be  ease  of  reading  the  subject  interview  section. 

Chi-Squares.  When  analyzing  results  of  Part  IV  of  the  questionnaire,  Chi-Squares 
were  used  for  determining  the  statistical  significance  of  the  comparative  data  from 
respondents  reading  the  high-impact  and  low-impact  reports.  Once  again,  the  dependent 
variables  were  the  specific  factors  coded  into  the  questions  themselves,  while  the 
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independent  variable  in  all  tests,  was  the  specific  report  writing  style  ( Le .,  high-impact  or 
low-impact). 

Of  the  ten  questions  in  Part  IV,  eight  were  analyzed  using  this  technique. 
Questions  6  and  8  were  not  comparative  (Le.,  choosing  between  Case  I  or  Case  II),  but 
were  yes/no  questions.  Because  of  this  yes/no  design,  there  was  no  way  to  tie  these 
questions  to  the  independent  variable,  which  was  the  effect  of  the  writing  style. 


Organization  of  Analysis 

The  analysis  of  each  part  of  the  questionnaire  was  to  determine  the  statistical 
significance  between  the  question  responses  for  high-impact  versus  low-impact  ROIs.  The 
key  difference  throughout  the  analysis  was  the  stratification  of  the  adjudicator  population 
and/or  the  specific  case  treatment  analyzed.  This  breakdown  is  as  follows: 

•  Both  Cases:  The  high/low  impact  Czamek  and  Rokitka  high/low-impact  cases 
were  analyzed  using  the  entire  population  of  adjudicators. 

•  Czamek  Cases:  The  high/low-impact  Czamek  cases  were  analyzed  using  the  entire 
population  of  adjudicators. 

•  Rokitka  Cases:  The  high/low-impact  Rokitka  cases  were  analyzed  using  the  entire 
population  of  adjudicators. 

•  Army  Adjudication  Site:  The  high/low  impact  Czamek  and  Rokitka  high/low- 
impact  cases  were  analyzed  using  the  result  from  just  the  Army  population  of 
adjudicators. 

•  Navy  Adjudication  Site:  The  high/low  impact  Czamek  and  Rokitka  high/low- 
impact  cases  were  analyzed  using  the  result  from  just  the  Navy  population  of 
adjudicators. 

•  DISCR  Adjudication  Site:  The  high/low  impact  Czamek  and  Rokitka  high/low- 
impact  cases  were  analyzed  using  the  result  from  just  the  DISCR  population  of 
adjudicators. 

•  Demographic  Breakdown:  The  high/low  impact  Czamek  and  Rokitka  high/low- 
impact  cases  were  analyzed  using  a  demographic  breakdown  of  the  adjudicator 
population.  A  more  detailed  description  of  the  demographic  breakdown  is  given 
below. 
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Description  of  Demographic  Breakdown 

The  questionnaires  were  analyzed  using  a  demographic  breakdown  of  the 
adjudicator  population.  The  following  four  demographic  categories  were  used  for  this 
analysis: 

•  Age:  The  first  demographic  category  was  age,  young  and  old.  Young  was 
considered  below  one  standard  deviation  from  the  mean  adjudicator  age,  while  old 
was  everyone  else.  The  mean  age  of  adjudicators  was  44.2  years,  one  standard 
deviation  was  10.4  years,  so  the  cut-off  for  the  young  category,  was  33.8  years  old. 
Under  this  criterion,  3  adjudicators  fell  into  the  young  category,  while  36  fell  into 
the  old. 

•  Sex:  The  second  category  was  sex,  male  and  female.  The  sexual  breakdown  was  9 
male  to  31  female  adjudicators. 

•  Grade  Level:  The  third  category  was  grade  level.  The  population  was  stratified 
between  GS-11  versus  GS-12  and  GS-13  adjudicators.  Under  this  criterion,  9  fell 
into  the  GS-11  category,  28  into  the  GS-12  and  GS-13  category,  and  3  adjudicators 
did  not  list  their  grade  level  on  the  questionnaire. 

•  Years  in  Security  Administration  Service:  The  final  category  was  the  number  of 
years  of  service  that  an  adjudicator  had  worked  in  security  administration.  The 
population  was  stratified  between  3  years  or  less  experience,  versus  greater  that  3 
years  experience.  Under  this  criterion,  3  adjudicators  fell  into  the  prior  category 
and  37  into  the  latter. 


Limitations 


The  sample  size  of  some  of  the  adjudicator  population  breakdowns  limited  the 
strength  of  the  analysis.  Specifically,  the  cell  sizes  were  n=6  when  we  compared  case 
treatment  and  decision  type  within  adjudication  sites,  and  as  small  as  n=3  for  two 
demographic  population  breakdowns.  Consequently,  the  data  in  these  report  sections 
must  be  interpreted  with  care. 

Also,  Suchan  tested  the  adjudicators  in  a  quiet  room,  free  from  distractions.  This 
environment  does  not  replicate  actual  workplace  conditions.  Adjudicators  work  in  busy 
offices  where  numerous  distractions  and  frequent  interruptions  can  make  reading  long 
ROIs  difficult.  Because  of  this  atypical  testing  environment,  the  questionnaire  results  do 
not  capture  all  of  the  high-impact  style’s  possible  reading  ease  and  ;nformation-retention 
advantages. 
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Presentation  and  Analysis  of  Adjudication  Times  and  the 
Adjudicators’  Impressions  of  Case  Information 


This  chapter  analyzes  the  times  adjudicators  required  to  assess  the  high-  and  low- 
impact  cases  and  their  responses  to  Questionnaire  Part  II  (see  Appendix  A).  The 
chapter  is  organized  in  three  sections;  the  first  two  sections  present  and  analyze  the 
adjudication  times  and  Questionnaire  Part  II  respectively,  while  the  final  section  draws 
conclusions. 


Presentation  and  Analysis  of  the  Time  Required 
to  Adjudicate  Both  the  High  and  Low  Cases 

These  results  indicate  that  there  were  no  statistically  significant  differences 
between  the  time  required  to  adjudicate  the  high-impact  Czamek  and  Rokitka  cases 
versus  their  low-impact  counterparts.  The  mean  times  to  adjudicate  both  the  high-  and 
low-impact  cases  are  18.33  and  20.08  minutes,  respectively.  The  corresponding  standard 
deviations  are  6.51  and  8.54  minutes.  The  resulting  value  of  the  F-statistic  is  1.046,  which 
shows  a  significance  of  31%.  This  significance  level  is  well  outside  of  the  5-10%  criteria. 

The  analysis  results  from  the  Czamek  and  Rokitka  cases,  each  adjudication  site, 
and  most  of  the  demographic  breakdowns,  also  produced  no  statistically  significant 
differences,  with  one  exception:  the  male  adjudicator  population. 

The  male  adjudicator  results,  however,  indicate  that  there  was  a  statistically 
significant  difference  between  the  time  male  adjudicators  required  to  adjudicate  the  high- 
impact  Czamek  and  Rokitka  cases  versus  the  low-impact  counterparts.  The  mean  times 
to  adjudicate  both  the  high-  and  low-impact  cases  are  13.5  and  21.75  minutes, 
respectively.  The  corresponding  standard  deviations  are  1.91  and  12.2  minutes.  The 
resulting  value  of  the  F-statistic  is  3.98,  which  shows  a  significance  of  6.6%  and  satisfies 
the  10%  confidence  criterion. 

Overall,  adjudication  testing  time  results  indicate  that  the  high-impact  design, 
organization  and  writing  style  does  not  significantly  decrease  the  time  required  to 
adjudicate  cases. 


Presentation  and  Analysis  of  Questionnaire  Part  II  Results 

This  section  presents  and  analyzes  the  results  of  the  questionnaire  which  tested  the 
adjudicators’  impressions  of  case  information. 

Questionnaire  Part  II  tests  adjudicators’  impressions  concerning  case  information. 
Bach  of  the  23  questions  in  this  part  of  the  survey  asked  adjudicators  to  rate  a  specific 
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aspect  concerning  the  presentation  of  case  information,  such  as  its  readability  and  the 
ease  of  finding  derogatory  information.  A  five-point  bipolar  scale  was  used  to  capture 
results.  Specifically,  this  part  of  the  survey  was  designed  to  determine  if  there  were 
significant  differences  between  the  response  to  the  questions  for  the  high-  and  low-impact 
cases. 


The  first  subsection  gives  direction  on  how  to  read  the  tables  of  results.  The  last 
draws  conclusions.  The  middle  subsections  are  organized  as  follows: 

•  Both  Cases:  Analysis  of  both  Czamek  and  Rokitka  high/low  impact  cases  using 
entire  adjudicator  population. 

•  Czamek  Cases:  Analysis  of  Czamek  high/low  impact  cases  using  entire  adjudicator 
population. 

•  Rokitka  Cases:  Analysis  of  Rokitka  high/low  impact  cases  using  entire  adjudicator 
population. 

•  Separate  Adjudication  sites:  Analysis  of  each  Adjudication  site  (Army,  Navy,  and 
DISCR)  using  both  cases. 

•  Male  Adjudicators:  Analysis  of  both  cases  using  Male  Adjudicators. 


Directions  for  Reading  Tables  of  Results 

There  are  seven  tables  of  results  in  this  chapter.  Each  table  has  a  title  block  and 
seven  columns  of  information.  Below  is  an  extract  of  the  title  block  and  column  heading 
with  an  explanation  of  each. 


SUMMARY  OF  QUESTIONNAIRE  PART  II  RESULTS 
CASES  AND  ADJUDICATORS  TESTED  (n  =  sample  size) 
ADJUDICATOR’S  IMPRESSIONS  OF  CASE  INFORMATION 


QUEST  # 

HI 

HI  S.D. 

LI 

LI  S.D. 

F-STAT 

SIG.  of  F 

MEAN 

MEAN 

The  title  block  is  the  same  for  all  tables  with  the  exception  of  the  second  line, 
which  is  highlighted  above.  This  line  states  which  cases  and  adjudicators  were  included 
for  the  analysis.  An  explanation  of  the  seven  columns  headings  follows: 

•  QUEST  #:  The  Part  II  question  number  analyzed. 
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HI  MEAN:  The  mean  score  on  a  five  point  bipolar  scale  for  the  high-impact  cases. 


•  HI  S.D.:  The  standard  deviation  of  the  high-impact  score. 

•  LI  MEAN:  The  mean  score  on  a  five  point  bipolar  scale  for  the  low-impact  cases. 

•  LI  S.D.:  The  standard  deviation  of  the  low-impact  score. 

•  F-STAT:  The  value  of  the  F  Statistic  resulting  from  the  ANOVA  testing. 

•  SIG.  of  F:  The  significance  of  the  F  Statistic,  which  is  the  probability  that  the 
resulting  F  value  is  the  due  to  a  random  occurrence. 

The  results  of  questions  that  fall  within  the  5%  confidence  interval  are  shaded 
while  the  results  that  meet  the  10%  interval  are  highlighted  in  bold  type. 


The  Analysis  of  Both  the  Czamek  and  Rokitka  Case  Results 

The  first  primary  research  question,  outlined  in  the  first  chapter,  asks 

•  Will  there  be  statistically  significant  differences  between  adjudicators’  impressions 
of  their  ability  to  process  case  information  presented  in  a  high-impact  ROI  versus  a 
low-impact  one? 

The  answer  to  that  question  is  given  in  Table  1.  Out  of  the  results  to  23  questions,  13 
were  significant  within  5%  while  another  four  were  significant  within  10%.  Those 
questions  that  met  the  5%  criteria  will  be  presented  in  the  following  subsections. 


Questions  Statistically  Significant  Within  the  5%  Confidence  Interval 
Question  1:  Coverage  of  Issues 

This  question  asks  to  what  extent  does  the  case  provide  good  to  poor 
coverage  of  issues  to  enable  the  adjudicator  to  make  a  high  quality  adjudication  decision. 
The  mean  score  for  the  high-impact  cases  (1.9)  falls  just  to  the  left  of  the  somewhat  good 
rating,  while  the  mean  score  (2.4)  for  the  low-impact  cases  lies  between  the  somewhat 
good  and  neutral  rating.  These  results  indicate  that  adjudicators  feel  that  the  high- 
impact  style  provides  more  thorough  coverage  of  the  case  issues  than  the  low-impact 
style. 


Since  each  case  contained  the  same  content,  the  design,  organization,  and 
style  of  the  high-impact  ROI  affected  adjudicators’  perception  of  the  document’s 
thoroughness. 
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TABLE  1 


Summary  of  Questionnaire  Part  II  Results 


Analysis  of  Both  Czamek  &  Rokitka  Cases  (n  =  40) 
Adjudicator’s  Impressions  of  Case  Information 


Question  2:  Presentation  of  Derogatory  Information 

This  question  asks  if  the  way  a  field  agent  presents  derogatory  information 
in  a  case  makes  it  easy  or  difficult  to  apply  the  adjudication  criteria  to  that  information. 
The  mean  score  for  the  high-impact  cases  (1.95)  lies  just  to  the  left  of  the  somewhat  easy 
rating,  while  the  low-impact  scores  (2.46)  falls  between  the  somewhat  easy  and  neutral 
rating.  The  results  show  that  the  high-impact  style  makes  it  easier  to  apply  the 
adjudication  criteria. 

Questions  5,  6,  and  7:  Perception  of  Field  Agents  Awareness 

Questions  5,  6  and  7  gauge  the  adjudicators’  perception  of  field  agents’ 
awareness  of  their  needs.  Specifically,  the  questions  asked  the  adjudicators,  based  on  the 
way  information  was  presented  in  the  cases,  did  they  feel  the  field  agent  seemed  to  be 
aware  or  unaware  of  the  following: 

•  Question  5:  The  information  needed  to  make  a  high  quality  adjudication  decision 

•  Question  6:  How  to  make  the  case  easier  to  read 

•  Question  7:  The  large  amounts  of  daily  reading  done  by  the  adjudicators 

The  high-impact  cases  mean  scores  are  1.73  for  question  5,  2.05  for  question  6,  and  2.03 
for  question  7.  These  scores  all  fall  within  the  area  of  the  "somewhat  aware"  rating 
category.  In  contrast,  the  low-impact  cases  mean  scores  are  2.18  for  question  5,  2.58  for 
question  6,  and  2.63  for  question  7.  These  results  lie  between  the  "somewhat  aware"  and 
"neutral"  rating  categories. 

Although  the  adjudicators  perception  of  the  field  agents  who  wrote  the 
cases  are  tested  much  more  thoroughly  in  Part  III,  the  results  of  these  three  questions 
point  out  that  document  design,  organization  and  writing  style  do  affect  readers’ 
perception  of  the  writer’s  awareness.  The  author  of  a  high-impact  style  document  is 
perceived  as  having  a  greater  awareness  of  the  reader’s  needs  than  the  author  of  a  low- 
impact  one. 


Question  8:  Evaluation  of  Narrative  Report  Writers 

Question  8  asked  adjudicators  if  they  believed  that  the  field  agents  who 
wrote  the  narrative  reports  for  the  cases  were  good  or  poor  writers.  The  high-impact 
mean  score  (1.93)  falls  just  to  the  left  of  the  "somewhat  good"  rating,  while  the  low- 
impact  score  (2.5)  lies  exactly  in  the  middle  of  the  "somewhat  good"  to  "neutral"  rating. 
Answers  to  this  question  showed  the  greatest  statistical  significance.  They  are  significant 
at  the  .6%  confidence  level.  These  results  show  that  the  majority  of  adjudicators  believe 
that  the  authors  of  the  high-impact  narrative  reports  are  better  writers  than  the  authors 
of  low-impact  reports. 
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Question  11,  and  12:  Ease  of  Finding  and 

Reading  Derogatory  Information 

Questions  11  and  12  tested  the  ease  of  finding  and  reading  derogatory 
information  in  the  cases.  The  high-impact  mean  scores  are  1.9  for  question  11  and  1.83 
for  question  12,  which  both  fall  to  the  left  of  the  somewhat  easy  rating.  The  low-impact 
mean  scores  are  2.5  for  question  11  and  2.33  for  question  12,  which  both  fall  between  the 
somewhat  easy  to  neutral  rating. 

These  results  indicate  that  adjudicators  found  the  high-impact  style  cases 
made  finding  and  reading  information  easier.  Some  of  the  specific  characteristics  of  a 
high-impact  document  that  make  finding  and  reading  information  easier  include  the  use 
of 

•  heading  and  subheadings 

•  lists  and  bullets 

•  internal  previews 

•  active  verbs 

•  explicit  transitional  tags 

The  results  show  that  these  and  other  high-impact  stylistic  strategies 
created  the  perception  that  finding  and  reading  derogatory  information  was  easier. 

Questions  14, 16,  and  19:  Ease  of  Remembering,  Locating 

and  Following  Presented  Case  Information 

These  questions  asked  the  adjudicators  the  relative  ease  they  had  in  the 
following  areas: 

•  Question  14:  remembering  the  presented  case  information 

•  Question  16:  locating  the  essential  information  necessary  for  the  efficient 
adjudication  of  the  case 

•  Question  19:  following  the  chronological  sequence  of  events  in  the  case 

The  high-impact  cases  mean  scores  are  1.9  for  question  14,  1.8  for  question  16,  and  2.05 
for  question  19.  These  scores  all  fall  in  the  area  of  the  "somewhat  easy"  rating  category. 
The  low-impact  cases  mean  scores  are  2.5  for  question  14,  2.48  for  question  16,  and  2.6 
for  question  19.  These  results  cluster  between  the  "somewhat  easy"  and  "neutral"  rating 
categories. 
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The  results  show  that  adjudicators  feel  that  information  presented  in  a 
high-impact  style  is  easier  to  remember  and  locate  than  information  presented  in  low- 
impact  style.  They  also  indicate  that  the  high-impact  style  makes  it  easier  to  follow 
chronological  sequences  of  events.  This  can  be  interpreted  to  mean  that  the  high-impact 
style  increases  information  retention  in  short-term  memory,  improves  efficiency  in 
locating  essential  information  in  the  document,  as  well  as  improves  their  ability  to  follow 
a  sequence  of  events. 

Question  15:  Ease  of  Adjudicating  Case 

The  adjudicators  were  asked  whether  they  found  it  easy  o»-  difficult  to 
adjudicate  the  case.  The  high-impact  mean  score  (1.78)  falls  just  to  the  left  of  the 
somewhat  easy  rating,  while  the  low-impact  score  (2.3)  lies  between  the  "somewhat  easy" 
to  "neutral"  rating. 

The  result  amplifies  the  finding  to  questions  14,  16,  and  19.  Ease  in 
locating  and  remembering  essential  case  information,  and  following  chronological  events 
should  lead  to  the  easier  adjudication  of  cases. 

Question  22:  Fatigue  of  Reading 

Question  22  asked  the  adjudicators  whether  they  agreed  or  disagreed  with 
the  statement,  "Reading  this  case  tired  me  out."  The  high-impact  mean  score  (4.38)  falls 
between  the  "somewhat  disagree"  and  "disagree"  rating  categories,  while  the  low-impact 
score  (3.78)  lies  between  the  "neutral"  rating  and  the  "somewhat  disagree"  ratings. 

The  above  results  indicate  that  the  high-impact  cases  caused  less  fatigue 
than  the  low-impact  ones.  The  high-impact  document  factors  that  may  contribute  to  this 
decreased  c'  tigue  include  the  use  of  headings  and  subheadings,  as  well  as  bullets  and 
lists. 


Questions  Statistically  Significant  within  the  10%  Confidence  Interval.  Although 
the  results  to  questions  3,  4,  18,  and  23  did  not  meet  the  5%  criterion,  they  did  show 
significance  within  the  10%  confidence  interval.  Questions  3  and  4  tested  the 
adjudicators’  confidence  in  the  quality  of  the  field  agents’  investigation  and  the  soundness 
of  their  adjudication  decision.  The  frequency  of  rereading  sentences  was  the  objective  of 
question  18,  while  eye  strain  was  the  factor  tested  for  in  question  23. 

The  high-impact  cases  mean  scores  are  1.93  for  question  3,  1.38  for  question  4, 

3.88  for  question  18,  and  3.7  for  question  23.  The  high-impact  scores  for  both  questions 
3  and  4  fall  between  the  "confident"  and  "somewhat  confident"  rating  category,  question 
18  scores  lie  between  the  "somewhat  never"  to  "never"  rating,  and  question  23  scores  fall 
between  the  "somewhat  disagree"  to  "disagree"  ratings. 

The  low-impact  cases  mean  scores  are  2.35  for  question  3,  1.65  for  question  4,  3.45 
for  question  18,  and  3.1  for  question  23.  The  low-impact  scores  for  questions  3  fall 
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between  the  "somewhat  confident"  and  "neutral"  rating  category,  question  4  falls  between 
the  "confident"  and  "somewhat  confident"  ratings,  question  18  scores  lie  between  the 
"somewhat  never"  to  "never"  rating,  and  question  23  scores  fall  between  the  "somewhat 
disagree"  to  "disagree*  ratings. 

In  all  of  the  above  questions,  it  is  inferred  that  the  design,  organization,  and  writing 
style  of  the  high-impact  cases  1)  increased  the  confidence  in  the  field  agent’s  investigation 
and  adjudicators’  decision,  2)  reduced  the  frequency  of  sentence  rereading,  and  3) 
reduced  eye  strain. 

Summary  of  Both  the  Czamek  and  Rokitka  Case  Results.  When  we  combine  the 
results  of  the  13  questions  that  passed  the  5%  confidence  test,  and  the  four  that  passed 
the  10%  test,  it  is  clear  that  adjudicators  feel  the  document  design,  organization,  and 
writing  style  of  the  high-impact  cases  to  be  superior  to  the  currently  used  low-impact  style 
cases.  As  will  be  seen  in  the  following  sections,  this  result  is  not  necessarily  true  when 
the  Czamek  and  Rokitka  cases  or  the  adjudication  sites  are  analyzed  separately. 

Analysis  of  the  High  and  Low-Impact  Czamek  Cases 

The  results  of  the  analysis  of  the  high-  and  low-impact  Czamek  cases  are 
presented  in  Table  2.  Out  of  the  23  questions  analyzed,  none  of  the  results  fell  within 
the  5%  confidence  interval  and  only  four  (#’s  2,  8,  16,  and  22)  met  the  10%  confidence 
criterion.  The  following  is  a  summary  of  each  of  the  four  questions: 

•  Question  2:  How  easy  was  it  to  applying  the  adjudication  criteria  to  the  case 

information? 

•  Question  8:  What  was  the  quality  (good  to  poor)  of  the  narrative  report  writers? 

•  Question  16:  How  easy  was  it  to  locate  essential  case  information? 

•  Question  22:  Was  reading  the  case  tiring? 

Comparing  the  responses  to  these  questions  with  those  from  the  analysis  of  both 
cases,  we  see  consistent  findings.  That  means  that  the  above  statistically  significant 
questions  were  also  found  significant  within  either  the  5%  or  10%  interval  in  the 
combined  Czarnek  and  Rokitka  analysis  (Table  1). 

Given  that  there  were  only  four  statistically  significant  questions  overall,  and  all  -f 
these  fell  only  within  the  10%  interval,  these  results  tell  us  that  there  was  not  a  strong 
difference  in  adjudicators’  impressions  of  the  high-  and  low-impact  Czamek  cases.  It  can 
be  further  inferred  that  Czamek  results  are  not  the  driving  force  behind  the  large 
amount  of  statistical  significance  found  when  both  the  Czamek  and  Rokitka  cases  were 
analyzed  together.  This  is  counterintuitive,  because  the  Czamek  case  content  is  more 
complicated  and  longer  than  the  Rokitka  case.  Common  sense  would  dictate  that  the 
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TABLE  2 


Summary  of  Questionnaire  Part  II  Results 
Analysis  of  the  Czamek  Case  (n  =  20) 
Adjudicator’s  Impressions  of  Case  Information 
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longer  and  more  complicated  a  document’s  content,  the  more  helpful  the  high-impact 
style  would  be  in  reading,  finding  and  locating  information. 

Analysis  of  the  High-  and  Low-Impact  Rokitka  Cases 

The  results  of  the  high-  and  low-impact  Rokitka  case  data  are  presented  in  Table 
3.  Questions  8,  12,  14,  and  19  were  found  to  be  statistically  significant  within  the  5% 
interval  while  questions  1,  5,  6,  7,  11,  15,  and  16  were  significant  within  the  10%  interval. 
The  following  is  a  summary  of  each  of  the  four  questions  that  met  the  5%  confidence 
criteria: 

•  Question  8:  What  was  the  quality  (good  to  poor)  of  the  narrative  report  writers? 

•  Question  12:  How  easy  was  it  to  read  the  derogatory  case  information? 

•  Question  14:  How  easy  was  it  to  remember  case  information? 

•  Question  19:  How  easy  was  it  to  follow  chronological  events  in  the  case? 

Questions  meeting  the  10%  criteria  asked  the  following:  question  1,  the  coverage  of 
issues  in  the  case;  question  5  and  7,  the  field  agents’  awareness  of  the  adjudicators’ 
information  needs  and  daily  reading  requirements;  question  6,  the  field  agents’  ability  to 
make  the  case  easier  to  read;  and  finally,  questions  11,  15  and  16:  the  ease  in  locating 
derogatory  and  essential  information  as  well  as  adjudicating  the  case. 

Comparing  the  results  of  these  questions  with  those  from  the  analysis  of  both 
cases,  we  see  consistent  findings.  In  other  words,  the  statistically  significant  response  to 
questions  listed  above  was  also  found  significant  within  either  the  5%  or  10%  confidence 
interval  in  the  combined  Czamek  and  Rokitka  analysis  (Table  1). 

Comparing  Rokitka  case  results  to  the  Czamek  (Table  3)  shows  that  the 
document  design,  organization,  and  writing  style  of  the  Rokitka  cases  had  a  greater 
statistical  significance  on  the  adjudicator’s  impressions  of  both  the  high-  and  low-impact 
writing  style  than  the  Czamek  cases.  This  comparison  makes  clear  that  the  Rokitka 
cases  are  the  driving  force  behind  the  statistically  significant  results  of  the  analysis  of  both 
cases. 
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TABLE  3 


Summary  of  Questionnaire  Part  II  Results 
Analysis  of  the  Rokitka  Case  (n  =  20) 


Adjudicator’s  Impressions  of  Case  Information 
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Analysis  of  the  Separate  Adjudication  Sites  using  both  the  High-  and  Low-impact 
Czamek  and  Rokitka  Cases 


This  section  presents  and  analyzes  the  separate  adjudication  site  results.  There  are 
three  subsections  in  this  section;  each  subsection  presents  and  analyzes  the  result  for  the 
Army,  Navy  or  DISCR  sites  respectively. 

Presentation  and  Analysis  of  the  Army  Adjudication  Site  Results.  Results  from  the 
Army  adjudication  site  produced  more  statistically  significant  results  than  the  Navy  and 
DISCR  sites  combined.  Table  4  shows  that  the  results  to  three  questions  (#’s  1,  4,  and 
12)  were  significant  within  the  5%  confidence  interval,  while  five  (#’s  6,  8,  10,  14,  and 
20)  were  within  10%.  The  following  is  a  summary  of  each  of  the  three  questions  that 
met  the  5%  confidence  interval  criteria. 

•  Question  1:  What  was  the  quality  (good  to  poor)  of  the  coverage  of  issues  in  the 

case? 

•  Question  4:  How  confident  was  your  adjudication  decision? 

•  Question  12:  How  easy  was  it  to  read  the  derogatory  case  information? 

Questions  meeting  the  10%  criteria  asked  about  the  following:  question  6,  the  field 
agents’  ability  of  make  the  case  easier  to  read;  question  8,  the  quality  of  narrative  report 
writing;  question  10,  the  perceived  speed  of  reading  the  case;  question  14,  the  ease  of 
remembering  derogatory  information  in  the  case;  and  finally,  question  20,  the  frequency 
of  underlining  or  note  taking. 

Comparing  the  above  results  with  those  from  the  analysis  of  both  cases  using  the 
entire  adjudicator  population,  we  see  three  key  differences.  The  first  two  are  that  the 
results  of  both  question  10  and  20  were  not  found  to  be  statistically  significant  when  both 
cases  were  analyzed.  The  third  difference  is  that  the  result  to  question  4  fell  within  the 
5%  confidence  interval  for  this  site  but  was  only  at  the  10%  level  in  Table  1. 

All  other  statistically  significant  results  from  the  Army  sites  are  consistent  with  the 
combined  case  analysis.  That  means  that  excluding  the  above  exceptions,  all  questions 
found  significant  within  either  the  5%  or  10%  confidence  interval  were  also  significant  in 
the  combined  Czamek  and  Rokitka  analysis  (Table  1). 

We  also  found  that  there  were  several  questions  in  the  Army  analysis  that  fell 
outside  the  significant  range  but  were  significant  in  the  analysis  of  both  cases.  These 
results  may  be  caused  by  the  smaller  sample  size  of  the  Army  adjudicator  test 
population. 

We  are  uncertain  why  the  Army  site  has  both  an  abundance  of  significant  results 
and  notable  statistical  inconsistences  with  the  results  of  the  combined  Czarnek  and 
Rokitka  cases. 
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TABLE  4 


Summary  of  Questionnaire  Part  II  Results 
Analysis  of  Army  Site  (n  =  12)  Using  Both  Cases 
Adjudicator’s  Impressions  of  Case  Information 


QUEST# 


HI 

MEAN 


HI  S.D. 


LI 

MEAN 


LI  S.D. 

F-STAT 

SIG.  of  F 

0.9 

4.61 

0.O4S 

1 

1.58 

0.224 

0.89 

2.33 

0.144 

3.79 


0.006 


3.05 


1.58 


4.77 


0.044 


0.24 


0.117 


.066 


0.94 


.097 


0.223 


0.041 


13 

2.17 

1.11 

2.67 

1.07 

1.2 

0.286 

14 

1.92 

0.9 

2.67 

1.23 

3.29 

■ 

.085 

15 

1.83 

0.94 

2.33 

1.07 

1.55 

0.227 

16 

2 

1.04 

2.5 

0.9 

1.48 

0.239 

17 

3.42 

1 

3.08 

1.38 

0.43 

0.517 

18 

3.75 

0.97 

3.42 

1.24 

0.5 

0.485 

19 

1.92 

0.9 

2.5 

1.31 

1.6 

0.22 

20 

3.67 

0.78 

3 

0.85 

3.72 

■ 

.068 

21 

4.67 

0.65 

4.33 

0.78 

1.38 

0.254 

22 

4.33 

0.78 

3.83 

1.11 

1.7 

0.207 

23 

3.83 

1.47 

3.25 

1.42 

0.953 

0.341 

Presentation  and  Analysis  of  the  Naw  Adjudication  Site  Results.  The  results  of 
the  Navy  site  analysis  are  presented  in  Table  5.  Only  the  results  for  question  #2  were 
statistically  significant  within  the  5%  confidence  interval,  and  four  (#’s  1,  7,  8,  and  16) 
were  within  the  10%  interval.  Question  2  asked  adjudicators  if  they  felt  it  was  easy  or 
difficult  to  apply  the  adjudication  criteria.  The  questions  meeting  the  10%  criteria  asked 
the  following:  question  1,  the  coverage  of  issues  in  the  case;  question  7,  the  field  agents 
awareness  of  the  adjudicators’  daily  reading  requirement;  question  8,  the  quality  of 
narrative  report  writing;  and  finally,  question  16,  the  ease  in  locating  essential  case 
information. 

Comparing  the  above  results  with  those  from  the  analysis  of  both  cases  using  the 
entire  adjudicator  population,  we  find  the  results  are  mostly  consistent.  The  only 
exceptions  are  that  there  are  fewer  statistically  significant  results  in  the  Navy  Site 
analysis,  and  questions  1,  7,  8  and  16  fell  within  the  10%  criteria  at  the  Navy  Site  but 
were  at  the  5%  level  when  treatment  of  both  cases  were  combined. 

Presentation  and  Analysis  of  the  DISCR  Adjudication  Site  Results.  The  result  of 
the  DISCR  site  analysis  are  presented  in  Table  6.  This  site  produced  the  least  amount  of 
statistically  significant  differences  (Le.,  two  questions).  The  results  to  one  question  (#22) 
were  statistically  significant  within  the  5%  confidence  interval  while  the  other  question 
(#23)  fell  within  the  10%  interval.  Questions  22  and  23  asked  the  adjudicators  if  reading 
the  case  caused  fatigue  and  eye  strain,  respectively.  Comparing  the  above  results  with 
those  from  the  analysis  of  both  cases  using  the  entire  adjudicator  population,  we  see  that 
the  results  are  consistent. 

We  must  note  that  the  results  to  the  other  21  questions  were  well  outside  the  10% 
interval.  From  this,  we  can  infer  that  there  is  relatively  little  difference  in  the  DISCR 
adjudicators’  impression  of  the  case  information  between  the  two  styles.  This  lack  of 
difference  is  far  greater  than  results  from  either  the  Army  or  Navy  Adjudication  Sites 

Presentation  and  Analysis  of  the  Male  Adjudicator  Results 

An  analysis  of  several  demographic  breakdowns  of  the  adjudicator  population  was 
performed.  As  with  the  results  of  the  analysis  of  the  adjudicating  times,  the  only  group 
to  produce  a  large  amount  of  statistically  significantly  results  was  the  male  adjudicators. 
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TABLE  5 


Summary  of  Questionnaire  Part  II  Results 
Analysis  of  Navy  Site  (n  =  12)  Using  Both  Cases 


Adjudicator’s  Impressions  of  Case  Information 


QUEST  # 

HI 

MEAN 

HI  S.D. 

LI 

MEAN 

LI  S.D. 

F-STAT 

SIG.  of  F 

1 

1.92 

0.79 

2.67 

1.23 

4.14 

jHW 

1.75 

31 

'  641 

0023 

3 

1.92 

0.9 

2.42 

0.9 

2.85 

0.108 

4 

1.33 

0.49 

1.33 

0.49 

0.013 

0.912 

5 

1.83 

0.72 

2.42 

1.08 

2.67 

0.119 

6 

2.25 

1.05 

2.67 

0.89 

1.1 

0.305 

7 

233 

1.07 

3.25 

136 

3.1 

mm 

8 

2 

2.67 

0.89 

3.27 

mm 

9 

3.42 

1.56 

3.5 

1 

0.023 

0.882 

10 

2.58 

1.16 

2.92 

1 

0.548 

0.468 

11 

2 

1.04 

2.5 

1.31 

0.989 

0.332 

12 

1.83 

0.83 

2.17 

1.03 

0.708 

0.412 

13 

2.33 

1.5 

2.42 

0.79 

0.027 

0.872 

14 

1.83 

0.94 

2.58 

1.24 

2.55 

0.126 

15 

1.67 

0.78 

2.42 

1.31 

2.68 

0.117 

16 

1.75 

m/m 

2.58 

138 

333 

■SSI 

17 

3.67 

1.07 

3.08 

0.89 

0.635 

0.435 

18 

3.92 

1.08 

3.42 

0.9 

0.954 

0.34 

19 

1.83 

1.11 

3.25 

1.44 

1.96 

0.177 

20 

2.58 

1.05 

3.33 

1.15 

0.033 

0.858 

21 

4.42 

0.9 

4.58 

0.67 

0.256 

0.618 

22 

4.25 

0.96 

4.08 

1.08 

0.159 

0.695 

23 

3.5 

1.31 

3.33 

1.56 

0.079 

0.782 
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TABLE  6 


Summary  of  Questionnaire  Part  II  Results 
Analysis  of  DISCR  Site  (n  =  12)  Using  Both  Cases 
Adjudicator’s  Impressions  of  Case  Information 


QUEST# 

HI 

MEAN 

HI  S.D. 

LI 

MEAN 

LI  S.D. 

F-STAT 

SIG.  of  F 

1 

1.75 

1.36 

2.08 

0.9 

0.24 

0.63 

2 

2 

1.55 

2.33 

1.15 

0.282 

0.602 

3 

1.67 

1.23 

12.17 

0.94 

0.78 

0.389 

4 

1.33 

1.15 

1.67 

0.98 

0.358 

0.556 

5 

1.5 

1.24 

1.92 

1.16 

0.438 

0.516 

6 

2 

1.48 

2.5 

1.38 

0.422 

0.422 

7 

1.75 

1.36 

2.25 

1.22 

0.826 

0.374 

8 

1.75 

1.22 

2.5 

1.24 

2.056 

0.167 

9 

3.5 

1.73 

3.92 

1.24 

0.475 

0.498 

10 

2.33 

1.44 

2.67 

1.37 

0.31 

0.584 

11 

1.58 

1.16 

2.42 

1.38 

2.34 

0.142 

12 

1.75 

1.22 

2.25 

1.14 

0.989 

0.332 

13 

2.17 

1.64 

2.58 

1.51 

0.41 

0.529 

14 

1.83 

1.4 

2.42 

1.31 

1.02 

0.325 

15 

1.67 

1.37 

2.25 

1.29 

1.05 

0.317 

16 

1.67 

1.23 

2.42 

1.38 

1.8 

0.195 

17 

3.75 

1.22 

3.42 

1.31 

0.408 

0.532 

18 

2.25 

0.97 

3.58 

1.31 

1.88 

0.185 

19 

2.33 

1.37 

2.67 

1.37 

0.328 

0.573 

20 

3.5 

1.31 

3.17 

1.4 

0.357 

0.557 

21 

4.92 

0.29 

4.67 

0.89 

0.918 

0.349 

22 

4.67 

wiim 

- :  162 

5455 

0.028 

23 

4 

1.28 

2.83 

1.8 

3.14 

0.092 
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These  results  are  presented  in  Table  7.  The  results  to  nine  questions  (#’s  1,  2,  3, 
6,  7,  11,  14,  15,  16,  and  18)  were  found  to  be  statistically  significant  within  5%,  and  five 
question  (#’s  5,  8,  12,  17,  and  19)  within  10%.  The  following  is  a  summary  of  the  nine 
question  that  met  the  5%  confidence  interval  criteria: 

•  Question  1:  What  was  the  quality  (good  to  poor)  of  the  coverage  of  issues  in  the 
case? 

•  Question  2:  How  easy  was  it  to  apply  the  adjudication  criteria  to  the  case 
information? 

•  Question  3:  How  confident  were  you  in  the  quality  of  the  field  agent’s 
investigation? 

•  Question  6:  How  aware  were  field  agents  in  making  the  case  easier  to  read? 

•  Question  7:  How  aware  was  the  field  agent  to  the  adjudicator  daily  reading 
requirement? 

•  Question  11:  How  easy  was  it  to  find  derogatory  information? 

•  Question  14:  How  easy  was  it  to  remember  derogatory  information? 

•  Question  15:  How  easy  was  it  to  adjudicate  the  case? 

•  Question  16:  How  easy  was  it  to  locate  essential  case  information? 

•  Question  18:  How  often  were  sentences  reread? 

The  questions  meeting  the  10%  criteria  asked  the  following:  question  5,  the  field  agents’ 
awareness  of  the  adjudicators’  information  needs;  question  8,  the  quality  of  narrative 
report  writing;  question  12,  the  ease  of  reading  derogatory  information  in  the  case; 
question  17,  the  frequency  of  reading  section  in  the  case;  and  finally,  question  19,  the 
ease  of  following  the  chronological  sequence  of  events  in  the  case. 

Comparing  the  above  results  with  those  from  the  analysis  of  both  cases  using  the 
entire  adjudicator  population,  we  once  again  see  that  the  results  are  consistent.  These 
results,  however,  are  quite  surprising  considering  the  small  sample  size  (n=9)  of  male 
adjudicators. 
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TABLE  7 


Summary  of  Questionnaire  Part  II  Results 
Analysis  of  Male  Adjudicators  (n  =  9)  Using  Both  Cases 
Adjudicator’s  Impressions  of  Case  Information 
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Chapter  Conclusion 


The  results  from  the  high-  and  low-impact  treatment  of  the  Czamek  and  Rokitka 
cases  point  out  that  there  is  no  statistical  evidence  to  show  that  the  high-impact 
decreases  adjudication  time.  Conversely,  there  is  evidence  that  adjudicators  feel  that  the 
high-impact  style  is  an  improvement  over  the  low-impact  one.  Specifically,  adjudicators 
believe  that  the  high-impact  style 

•  provides  better  coverage  of  issues 

•  makes  it  easier  to  apply  adjudication  criteria  to  the  case  information 

•  makes  the  field  agents  who  wrote  the  case  seem  more  aware  of  1)  how  to  make 
the  case  easier  to  read,  2)  their  informational  needs,  3)  the  extensive  amount  of 
their  daily  reading 

•  makes  the  field  agent  seem  like  a  better  writer 

•  makes  the  derogatory  information  in  the  narrative  report  easier  to  find,  read, 
remember  and  adjudicate 

•  makes  following  the  chronological  sequence  of  events  in  the  case  easier  to  follow 

•  causes  less  fatigue  while  reading 

To  a  lesser  degree,  adjudicators  believe  that  the  high-impact  style  makes  them 
more  confident  in  the  soundness  and  quality  of  their  adjudication  decision,  causes  them 
to  reread  sentences  less  often,  and  produces  less  eye  stain. 

We  also  found  that  there  were  greater  statistically  significant  differences  in  the 
adjudicators’  impressions  of  the  case  information  in  the  Rokitka  cases  than  the  Czamek 
cases.  This  is  contrary  to  expected  results  because  the  Czamek  case  content  is  longer  and 
more  complicated  than  the  Rokitka  case.  One  would  expect  that  the  longer  and  more 
complicated  the  document’s  content,  the  more  helpful  the  high-impact  style  would  be  in 
reading,  finding,  and  locating  information. 

When  the  analysis  was  divided  by  adjudication  site,  the  statistical  significance  was 
not  nearly  as  great  as  the  result  from  the  entire  population.  However,  this  could  be 
attributed  to  smaller  sample  sizes.  The  greatest  statistical  differences  were  found  from 
the  analysis  of  the  Army  adjudication  site  data.  The  Navy  adjudication  site  data 
produced  the  second  most  statistically  significant  result,  while  DISCR  data  had  the  least. 

The  most  surprising  result  came  from  analyzing  the  demographic  breakdown  of 
adjudicators.  The  male  adjudicators  found  that  the  high-impact  style  decreased  the  time 
required  to  adjudicate  a  case.  They  also  felt  that  the  high-impact  cases  were  a  much 
greater  improvement  over  the  current  low-impact  cases  than  their  female  counterparts. 


39 


much  greater  improvement  over  the  current  low-impact  cases  than  their  female 
counterparts. 
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Presentation  and  Analysis  of  Part  III  Results: 

The  Adjudicator’s  Perception  of  the  Field  Agent 

Questionnaire  Part  III  (see  Appendix  B)  tests  adjudicators’  perception  of,  or 
feeling  towards,  the  field  agents  who  wrote  either  the  high-impact  or  low-impact  ROI. 
This  survey  section  asked  adjudicators  to  rate  the  field  agents  using  15  bipolar  adjectives, 
such  as  objectivity,  clarity  of  thinking,  reliability,  etc..  A  five-point  bipolar  scale  was  used 
to  capture  the  adjudicators’  feelings.  Specifically,  this  pa.t  of  the  survey  was  designed  to 
determine  if  the  high-  or  low-impact  style  caused  differences  in  adjudicators’  perceptions 
of  field  agents. 

This  chapter  presents  and  analyzes  the  results  of  Questionnaire  Part  III.  The  first 
section  of  the  chapter  gives  a  very  brief  explanation  of  how  to  read  the  tables  of  results, 
and  the  last  draws  conclusions.  The  middle  sections  cover  the  analysis  results  from  both 
cases,  the  separate  adjudication  sites,  and  finally  the  male  adjudicators. 


Directions  for  Reading  Tables  of  Results 

As  in  the  previous  chapter,  there  are  seven  tables  of  results.  These  tables  follow 
the  same  format  as  the  previous  tables  ( Le .,  title  block,  column  headings,  shading  of  5% 
results,  and  bolding  of  10%  results).  The  only  differences  are  the  information  in  the  title 
block,  the  results  to  the  specific  questions,  and  total  number  of  questions  (15  versus  23). 


Analysis  of  Both  Czarnek  and  Rokitka  Cases 

This  report’s  second  primary  research  question  asks 

•  Will  there  be  statistically  significant  differences  between  adjudicators’  perceptions 
or  feelings  towards  the  field  agents  who  wrote  a  high-impact  ROI  as  compared  to 
those  who  wrote  the  low-impact  ROI? 

The  answer  to  that  question  is  given  in  Table  8.  Out  of  15  bipolar  descriptive  adjectives, 
only  two  were  significant  at  5%  level,  while  four  were  significant  at  the  10%  level. 


Questions  Statistically  Significant  Within  the  5%  Confidence  Interval 

Question  8:  Agent  Parity  of  Thinking.  This  bipolar  adjective  determined  whether 
adjudicators  perceive  differences  in  the  clarity  of  thinking  between  the  field  agents  who 
wrote  the  high-  or  low-impact  case  treatments.  The  high-impact  mean  score  (1.9)  falls 
just  to  the  left  of  the  somewhat  clear  thinking  rating,  while  the  low-impact  score  (2.25) 
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TABLE  8 


Summary  of  Questionnaire  Part  III  Results 
Analysis  of  Both  Czamek  &  Rokitka  Cases  (n  =  40) 
Adjudicator’s  Perception  of  Field  Agent  Writing  ROI 


lies  between  the  somewhat  clear  thinking  and  the  neutral  rating.  The  results  indicate 
that  the  adjudicators  felt  the  writer  of  the  high-impact  ROI  is  a  more  lucid  thinker  than 
his  low-impact  counterpoint.  This  result  points  out  that  the  ROI  document  design, 
organization  and  writing  style  projects  a  more  intelligent  image  of  the  writer,  which  could 
increase  the  writer’s  credibility  with  the  reader. 

Question  10:  Efficiency  of  the  Field  Agent.  This  bipolar  adjective  determined 
whether  adjudicators  perceived  differences  in  efficiency  between  the  field  agents  who 
wrote  the  high-  or  low-impact  cases.  The  high-impact  mean  score  (1.85)  falls  just  to  the 
left  of  the  somewhat  efficient  rating,  while  the  low-impact  score  (2.3)  lies  between  the 
somewhat  efficient  and  the  neutral  rating. 
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The  results  show  that  adjudicators  perceive  the  high-impact  writer  as  more  efficient 
than  the  low-impact  one.  This  perception  of  efficiency  could  also  increase  the  field 
agent’s  credibility  with  the  adjudicators.  In  other  words,  the  more  efficient  the  field 
agent  seems  to  the  adjudicators,  the  less  likely  they  will  feel  that  the  Personnel  Security 
Investigation  (PSI)  was  conducted  in  a  shoddy,  haphazard  manner.  These  results  also 
suggest  that  adjudicators  view  the  document  factors  in  the  high-impact  style  as  a  more 
efficient  means  of  conveying  derogatory  information  than  the  low-impact  style. 


Questions  Statistically  Significant  Within  the  10%  Interval 

The  results  to  the  rating  of  the  bipolar  adjectives  5,  6,  7,  and  14  were  found  to  be 
significant  within  the  10%  confidence  interval.  These  bipolar  adjectives  determined 
whether  the  adjudicators  perceived  differences  in  the  field  agents  in  the  areas  of 
organization,  precision,  care,  and  judgment,  respectively. 

The  high-  and  low-impact  scores  and  their  relative  position  on  the  rating  scale  are 
as  follows: 

•  #5:  The  high-impact  score  (1.97)  falls  just  to  the  left  of  the  "somewhat  organized" 
rating.  The  low-impact  score  (2.4)  lies  between  the  "somewhat  organized"  and 
"neutral"  rating. 

•  #6:  The  high-impact  score  (1.98)  falls  just  to  the  left  of  the  "somewhat  precise" 
rating.  The  low-impact  score  (2.33  lies  between  the  "somewhat  precise"  and 
"neutral"  rating. 

•  #7:  The  high-impact  score  (1.83)  falls  to  the  left  of  the  "somewhat  careful"  rating. 
The  low-impact  score  (2.21)  lies  between  the  "somewhat  careful"  and  "neutral" 
rating.  This  results  shows  a  significance  almost  within  the  5%  confidence  interval 
(i.e.  5.5%) 

•  #14:  The  high-impact  score  (1.74)  falls  to  the  left  of  the  "fairly  sound  judgment" 
rating.  The  low-impact  score  (2.4)  lies  between  the  "fairly  sound  judgment"  and 
"neutral"  rating. 

Although  these  results  are  not  as  conclusive  as  the  findings  in  the  5%  interval,  they 
indicate  that  the  writers  of  high-impact  ROIs  are  regarded  as  more  organized,  precise, 
and  careful,  and  possess  better  judgment  than  their  low-impact  counterparts. 


Summary  of  Both  the  Czarnek  and  Rokitka  Case  Results 

When  we  combine  the  results  from  the  rating  of  the  two  bipolar  adjectives  that 
passed  the  5%  confidence  test,  and  the  four  that  passed  the  10%  test,  it  is  fairly  clear 
that  adjudicators  perceive  the  authors  of  the  high-impact  style  cases  to 
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•  possess  better  judgment 

•  be  more  organized 

•  be  more  precise 

•  be  more  careful 

•  be  more  clear  thinking 

•  be  more  efficient 

• 

than  the  writers  of  low-impact  style  cases.  These  perceived  attributes  may  increase  the 
field  agents’s  credibility  in  the  eyes  of  the  adjudicators  and  thus  produce  greater 
confidence  in  the  ROI  case  information.  This  increased  credibility  and  confidence  in  the 
case  information  would  reduce  the  requests  for  additional  information  concerning  the 
case  subject.  Taking  that  argument  one  step  further,  we  could  say  that  credibility 
improves  the  efficient  flow  of  information  in  the  adjudication  process  and  will  thus 
decrease  case  processing  time. 

As  will  be  seen  in  the  following  sections,  these  differences  in  perception  are  not 
necessarily  true  when  the  Czamek  and  Rokitka  cases  or  the  adjudications  sites  are 
analyzed  separately. 


Analysis  of  the  High-  and  Low-impact  Czamek  Cases 

The  analysis  results  of  Part  III  of  the  questionnaire  using  just  the  high-  and  low- 
impact  Czamek  cases  are  presented  in  Table  9.  The  analysis  of  the  15  ratings  of  the 
bipolar  adjectives  showed  that  none  of  the  results  was  found  to  be  statistically  significant 
within  either  the  5%  or  10%  confidence  intervals. 

These  results  tell  us  that  there  was  no  difference  in  adjudicators’  perception  of  the 
field  agents  who  wrote  either  the  high-  or  low-impact  Czamek  cases.  We  can  also  infer 
that  these  results  are  not  the  driving  force  behind  the  statistically  significant  results  to  the 
bipolar  ratings  when  both  the  Czamek  and  Rokitka  cases  were  analyzed  together.  Like 
the  Part  II  results,  this  lack  of  significant  difference  is  unexpected  because  the  Czamek 
case  content  is  much  more  complicated  and  longer  than  the  Rokitka  case  content.  One 
would  assume  that  writers  of  high-impact  documents  containing  longer  and  more 
complicated  information  would  be  more  positively  perceived  than  writers  of  low-impact 
ones. 


44 


TABLE  9 


Summary  of  Questionnaire  Part  III  Results 
Analysis  of  the  Czamek  Case 
Adjudicator’s  Perception  of  Field  Agent  Writing  ROI 


QUEST# 

HI 

MEAN 

HI  S.D. 

LI 

MEAN 

LI  S.D. 

F-STAT 

SIG.  of  F 

1 

1.85 

0.93 

2.3 

1.03 

0.846 

0.364 

2 

1.95 

1.08 

2 

0.97 

0.03 

0.863 

3 

1.58 

0.77 

1.75 

0.85 

0.144 

0.706 

4 

2.1 

1.07 

2.1 

1.07 

0.308 

0.582 

5 

2.12 

1.11 

2.3 

0.98 

0.282 

0.599 

6 

2.1 

1.21 

2.35 

0.81 

1.287 

0.264 

7 

1.85 

0.88 

2.21 

0.71 

1.459 

0.235 

8 

2.16 

1.12 

2.25 

0.85 

0.706 

0.407 

9 

2 

1.26 

2.2 

0.95 

0.172 

0.681 

10 

1.89 

1.05 

2.3 

0.92 

1.743 

0.195 

11 

2 

1.15 

2.15 

0.74 

0.085 

0.772 

12 

1.95 

1.03 

2.11 

0.88 

0.021 

0.885 

13 

2.05 

1.23 

2.2 

0.77 

0.007 

0.935 

14 

1.89 

1.1 

2.25 

0.97 

0.843 

0.365 

15 

2.3 

1.08 

2.25 

0.79 

0.1 

0.754 

Analysis  of  the  High-  and  Low-impact  Rokitka  Cases 

The  results  of  Part  III  using  just  the  high-  and  low-impact  Rokitka  case  data  are 
presented  in  Table  10.  Only  the  bipolar  rating  for  clarity  of  thinking  (#8)  was  found  to 
be  statistically  significant  within  the  5%  confidence  interval.  Three  other  bipolar  ratings 
of  organization  (#5),  efficiency  (#10),  and  judgment  (#14)  were  significant  within  the 
10%  interval. 

Comparing  these  semantic  differential  results  with  those  from  our  analysis  of  both 
cases,  we  see  consistent  findings.  The  statistically  significant  responses  described  above 
were  also  found  significant  within  either  the  5%  or  10%  confidence  interval  in  the 
combined  Czamek  and  Rokitka  analysis  (Table  8). 
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TABLE  10 


Summary  of  Questionnaire  Part  III  Results 
Analysis  of  the  Rokitka  Case 
Adjudicator’s  Perception  of  Field  Agent  Writing  ROI 


QUEST# 

HI 

MEAN 

HI  S.D. 

LI 

MEAN 

LI  S.D. 

F-STAT 

SIG.  of  F 

1 

1.7 

1.03 

2 

1.03 

0.851 

0.362 

2 

1.65 

0.75 

2.05 

1.05 

1.93 

0.173 

3 

1.5 

0.69 

1.8 

0.77 

1.693 

0.201 

4 

1.75 

0.85 

2.05 

0.76 

1.385 

0.247 

5 

1.85 

1.14 

2.5 

1.24 

2.998 

0.091 

6 

1.85 

0.93 

2.32 

1.11 

2.023 

0.163 

7 

1.8 

0.83 

2.2 

1.06 

2.352 

0.134 

8 

L65 

IBlliSB 

0.97 

5.577 

0.024 

9 

2 

1.26 

2.35 

1.27 

1.09 

0.303 

10 

1.8 

1.01 

23 

0.98 

337 

0.079 

11 

1.55 

0.76 

1.84 

0.83 

1.31 

0.26 

12 

1.55 

0.76 

2 

0.92 

1.884 

0.178 

13 

1.7 

0.8 

2.1 

1.25 

1.089 

0.303 

14 

1.6 

0.75 

2.2 

1.06 

3.219 

0.081 

15 

2 

1.03 

2.3 

1.26 

0.211 

0.648 

Unlike  the  Czamek  cases,  we  can  say  that  the  Rokitka  case  responses  are  the 
driving  force  behind  the  statistically  significant  results  we  found  when  both  cases  were 
combined. 


Analysis  of  the  Separate  Adjudication  Sites  Using 

Both  the  High-and  Low-impact  Czamek  and  Rokitka  Cases 

This  section  presents  and  analyzes  the  Part  III  results  for  each  separate 
adjudication  site.  This  section  will  be  presented  in  three  subsections:  Army,  Navy  and 
DISCR  results. 


46 


Presentation  and  Analysis  of  the  Army  Adjudication  Site  Results 

Unlike  the  last  chapter,  the  Army’s  Part  III  results  did  not  dominate  the  other  sites 
in  statistically  significant  results.  Table  11  shows  that  the  result  to  the  judgment  bipolar 
rating  (#14)  was  the  only  result  significant  to  within  either  the  5%  or  10%  confidence 
interval.  This  rating  is  consistent  with  the  combined  case  analysis. 


TABLE  11 


Summary  of  Questionnaire  Part  III  Results 
Analysis  of  Army  Site  (n  =  12)  Using  Both  Cases 
Adjudicator’s  Perception  of  Field  Agent  Writing  ROI 


QUEST  # 

HI 

MEAN 

HI  S.D. 

LI 

MEAN 

LI  S.D. 

F-STAT 

SIG.  of  F 

1 

2 

1.04 

2.17 

1.03 

0.149 

0.703 

2 

2 

0.85 

2.33 

0.89 

0.889 

0.357 

3 

1.75 

0.97 

2.08 

1.08 

0.597 

0.449 

4 

2.25 

0.75 

2.33 

0.98 

0.052 

0.823 

5 

2.08 

0.79 

2.58 

0.9 

2.093 

0.163 

6 

2.17 

1.03 

2.5 

1.17 

0.986 

0.333 

7 

2.17 

0.94 

2.55 

0.93 

0.86 

0.365 

8 

2 

0.85 

2.42 

1 

2.25 

0.15 

9 

2.67 

1.56 

2.42 

1.24 

0.048 

0.828 

10 

2.33 

1.07 

2.42 

1 

0.239 

0.63 

11 

1.83 

0.94 

2.25 

0.97 

2.05 

0.168 

12 

1.92 

0.9 

2.36 

0.92 

1.66 

0.213 

13 

2 

0.95 

2.33 

1.07 

1.181 

0.291 

14 

1.83 

0.94 

2.42 

1 

3.4 

0.081 

15 

2.08 

1.08 

2.42 

1.08 

1.014 

0.327 

47 


Presentation  and  Analysis  of  the  Naw  Adjudication  Site  Results 


The  results  of  the  Navy  site  analysis  are  presented  in  Table  12.  Only  the 
"precision"  bipolar  rating  (#6)  was  found  to  be  statistically  significant  within  the  5% 
confidence  interval.  Three  other  bipolar  ratings  for  clarity  of  thinking  (#8),  awareness 
(#9),  and  efficiency  (#10)  were  within  the  10%  interval.  Comparing  these  results  with 
those  from  the  analysis  of  both  cases,  we  see  consistent  findings,  with  only  one  exception. 
Question  6  was  statistically  significant  within  the  10%  interval  in  the  previous  analysis, 
but  the  Navy  results  show  it  falling  within  the  5%  interval. 


TABLE  12 


Summary  of  Questionnaire  Part  III  Results 
Analysis  of  Navy  Site  (n  =  12)  Using  Both  Cases 
Adjudicator’s  Perception  of  Field  Agent  Writing  ROI 


QUEST 

# 

HI  MEAN 

HI  S.D. 

LI  MEAN 

LI  S.D. 

F-STAT 

SIG.  of  F 

1 

1.67 

0.78 

2.17 

0.72 

1.005 

0.329 

2 

1.82 

0.6 

1.83 

0.72 

0.162 

0.692 

3 

1.55 

0.52 

1.83 

0.58 

0.722 

0.407 

4 

1.75 

0.75 

2.33 

0.98 

0.341 

0.567 

5 

2 

0.82 

2.25 

0.97 

0.278 

0.604 

MM 

1.83  : 

2M 

5,918 

0.026 

7 

1.67 

0.49 

2 

0.74 

2.033 

0.172 

8 

2 

1 

2-33 

0.98 

3317 

0.086 

9 

1.67 

0.78 

2.42 

1 

3.938 

0.064 

10 

1.73 

0.65 

2.42 

1.08 

3.938 

0.064 

11 

1.91 

0.94 

2 

0.77 

0.103 

0.752 

12 

1.73 

0.65 

2.08 

0.9 

0.054 

0.819 

13 

2 

1.13 

2.25 

1.06 

0.08 

0.781 

14 

1.83 

0.72 

2.33 

1.15 

0.095 

0.761 

15 

2.42 

1 

2.75 

1.06 

0.006 

0.937 
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Presentation  and  Analysis  of  the  DISCR  Adjudication  Site  Results 

The  DISCR  results  are  presented  in  Table  13.  This  site  produced  no  statistically 
significant  results.  Oddly  enough,  none  of  the  results  of  the  analysis  was  even  close  to 
falling  within  the  statistically  significant  range,  indicating  there  is  relatively  little  difference 
in  DISCR  adjudicators’  perception  of  the  field  agents  who  wrote  either  the  high-  or  low- 
impact  style  cases.  The  reason  why  the  adjudicators  show  no  preference  between  the 
two  cases  may  lie  in  the  unique  organizational  norms  at  the  DISCR. 


TABLE  13 


Summary  of  Questionnaire  Part  III  Results 
Analysis  of  DISCR  Site  (n  =  12)  Using  Both  Cases 
Adjudicator’s  Perception  of  Field  Agent  Writing  ROI 


QUEST 

# 

HI  MEAN 

HI  S.D. 

LI  MEAN 

LI  S.D. 

F-STAT 

SIG.  of  F 

1 

1.67 

1.15 

2.08 

1.44 

0.367 

0.552 

2 

1.58 

1.24 

1.92 

1.38 

0.213 

0.649 

3 

1.33 

0.49 

1.42 

0.67 

0.046 

0.833 

4 

1.58 

1.16 

1.83 

0.94 

0.165 

0.689 

5 

1.82 

1.6 

2.33 

1.5 

0.628 

0.438 

6 

1.67 

1.15 

2 

0.74 

0.667 

0.424 

7 

1.58 

0.9 

2.17 

1.11 

1.815 

0.193 

8 

1.67 

1.15 

2.08 

0.9 

0.969 

0.337 

9 

1.67 

1.15 

2.17 

1.27 

0.978 

0.334 

10 

1.5 

1.17 

2.08 

0.9 

1.788 

0.196 

11 

1.5 

1.17 

1.83 

0.72 

0.407 

0.531 

12 

1.5 

1.17 

1.83 

0.94 

0.338 

0.568 

13 

1.67 

1.15 

1.83 

1.11 

0.032 

0.86 

14 

1.55 

1.21 

2 

0.95 

0.877 

0.361 

15 

1.92 

1.16 

1.75 

0.86 

0.383 

0.543 
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Presentation  and  Analysis  of  the  Male  Adjudicator  Results 

Just  as  in  the  previous  chapter,  the  analysis  of  the  several  demographic 
breakdowns  produced  no  significant  results  except  for  the  male  adjudication  population. 
These  result  are  presented  in  Table  14.  The  bipolar  ratings  for  organization  (#5),  clarity 
of  thinking  (#8),  and  efficiency  (#10)  were  found  to  be  statistically  significant  at  the  5% 
level,  while  the  rating  for  thoroughness  (#1),  precision  (#6),  care  (#7),  confidence 
(#11),  and  judgment  (#14)  were  statistically  significant  at  the  10%  level. 


TABLE  14 

Summary  of  Questionnaire  Part  II  Results 
Analysis  of  Male  Adjudicators  (n  =  9)  Using  Both  Cases 
Adjudicator’s  Impressions  of  Case  Information 


For  the  most  part,  these  results  are  consistent  with  those  found  in  the  analysis  of 
both  cases  using  the  entire  population  (Table  8),  with  three  notable  exceptions.  There 
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are  two  additional  bipolar  ratings  (#  1  "thoroughness"  and  #11  "confidence")  with 
statistically  significant  results,  and  one  question  (#5  "clarity  of  thinking")  with  results  of 
greater  statistical  significance  (i.e.  5%  vice  10%).  Like  the  male  adjudicators’  results  in 
Part  II,  these  results  are  remarkable  considering  the  small  sample  size  (n=9). 


Chapter  Conclusion 

The  results  of  the  combined  Czamek  and  Rokitka  case  analysis  point  out  that 
there  is  statistically  significant  evidence  to  show  that  adjudicators  have  different 
perceptions  of  the  field  agents  who  wrote  high-impact  style  cases  versus  those  who 
composed  low-impact  ones.  Specifically,  they  feel  that  the  agents  writing  in  a  high-impact 
style  possess  better  judgment,  are  better  organized,  and  are  more  precise,  careful,  clear 
thinking,  and  efficient  than  writers  of  the  low-impact  cases.  These  perceived  attributes 
increase  field  agent  credibility,  which  could  improve  the  efficiency  of  the  entire 
adjudication  process. 

When  the  cases  were  analyzed  independently,  we  found  that  there  were  1)  no 
significant  differences  in  the  adjudicators’  perception  of  field  agents  who  wrote  the  high- 
or  low-impact  Czamek  cases;  and  2)  statistically  significant  differences  in  the  perceptions 
of  the  high-  and  low-impact  Rokitka  writers.  These  differences  were  found  in  the  areas 
of  organization,  clarity  of  thinking,  efficiency,  and  judgment. 

When  the  analysis  was  broken  down  by  adjudication  site,  the  statistical  significance 
was  not  nearly  as  great  as  the  results  from  the  entire  population.  This  result  could  be 
attributed  to  the  different  sample  sizes  used  in  the  analysis  of  both  cases  and  each  site 
(i.e.,  40  versus  12  adjudicators).  The  greatest  statistical  differences  were  found  in  the 
results  from  the  Navy  adjudicators,  with  the  Army  coming  in  a  distant  second.  DISCR 
came  in  third  with  no  statistically  significant  results. 

As  in  the  previous  chapter,  the  most  surprising  result  came  from  the  analysis  of  the 
demographic  breakdown  of  the  adjudicators.  The  survey  responses  from  the  male 
adjudicators  produced  statistically  significant  results. 
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Presentation  and  Analysis  of  Part  IV  Results: 
The  Writing  Style  Preferences  of  Adjudicators 


This  chapter  presents  the  results  from  Questionnaire  Part  IV  (see  APPENDIX  C). 
Part  IV  tested  the  adjudicators’  impressions  or  feelings  about  the  high-  and  low-impact 
writing  styles  by  having  adjudicators  choose  which  case  treatment  they  preferred  in 
relation  to  eight  factors.  These  preference  factors  include: 

•  presentation  of  information 

•  ease  of  reading,  remembering,  and  applying  the  adjudication  criteria 

•  amount  of  rereading 

•  confidence  in  adjudication  decision 

•  competence  of  the  field  agent  writing  the  case 

•  overall  writing  style 

The  chapter  is  organized  in  five  subsections.  The  first  subsection  gives  directions 
on  how  to  read  the  tables  of  results,  and  the  last  draws  conclusions.  The  remaining 
middle  subsections  are  organized  as  follows: 

•  Both  Cases:  Analysis  of  both  Czamek  and  Rokitka  high/low-impact  cases  using 
entire  adjudicator  population. 

•  Separate  Adjudication  sites:  Analysis  of  each  adjudication  site  (Army,  Navy,  and 
DISCR)  using  both  cases. 

•  Male  Adjudicators:  Analysis  of  both  cases  using  male  adjudicators. 


Direction  for  Reading  Tables  of  Results 

There  are  five  tables  of  results  in  this  chapter.  Each  table  has  a  title  block  and 
seven  columns  of  information.  Below  is  an  extract  of  the  title  block  and  column  heading 
with  an  explanation  of  each. 
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SUMMARY  OF  QUESTIONNAIRE  PART  IV  RESULTS  1 

CASES  AND  ADJUDICATORS  TESTED  (n  =  sample  size)  [ 

ADJUDICATOR’S  PREFERENCE  FOR  HIGH  OR  LOW-IMPACT  ROI  | 

QUEST  # 

PREFER  HI 

PREFER  LI 

CHI-SQR. 

STAT. 

SIG.  OF  1 

CHI-SQR.  I 
STAT.  | 

#  (%) 

#  (%) 

The  title  block  is  the  same  for  all  tables  with  the  exception  of  the  second  line, 
which  is  highlighted  above.  This  line  states  which  case  and  adjudicators  were  included  in 
the  analysis.  An  explanation  of  the  seven  column  headings  are  as  follows: 

•  QUEST  #:  The  Part  IV  question  analyzed,  excluding  question  6  and  8. 

•  PREFER  HI:  The  number  and  percentage  of  adjudicators  who  prefer  the  high- 
impact  style  cases  relative  to  the  tested  preference  factor. 

•  PREFER  LI:  The  number  and  percentage  of  adjudicators  who  prefer  the  low- 
impact  style  cases  relative  to  the  tested  preference  factor. 

•  CHI-SQR.  STAT.:  The  value  of  the  statistic  resulting  from  the  Chi-Square  testing. 

•  SIG.  of  CHI-SQR.  STAT.:  The  significance  of  the  chi-square  statistic,  which  is  the 
probability  that  the  chi-square  value  is  due  to  a  random  occurrence. 

The  results  of  questions  which  fall  within  the  5%  confidence  interval  are  shaded 
while  results  that  meet  the  10%  interval  are  highlighted  in  bold  type. 


Questions  6  and  8  were  posed  in  a  yes/no  format  and  could  not  be  statistically 
analyzed  using  Chi-squares.  The  answers  to  these  questions  are  included  in  the 
discussion  of  results. 


The  Analysis  of  Both  the  Czamek  and  Rokitka  Cases  Results 

The  third  primary  research  question,  outlined  in  the  first  chapter,  asks 

•  Will  there  be  statistically  significant  differences  in  adjudicators’  preferences  for  the 
case  information  presented  in  a  high-impact  versus  the  low-impact  style? 

The  answer  to  that  question  is  given  in  Table  15: 

Each  of  the  significant  results  in  the  table,  as  well  as  the  responses  to  yes/no 
questions,  will  be  compared  with  either  Part  II  or  III  survey  results.  Unless  otherwise 
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noted,  only  Part  II  or  III  results  significant  at  the  5%  or  10%  confidence  levels  are  used 
in  the  comparison. 


TABLE  15 


Summary  of  Questionnaire  Part  IV  Results 
Analysis  of  Both  Czamek  &  Rokitka  Cases  (n  =  40) 
Adjudicator’s  Preference  for  High  or  Low-impact  ROI 


QUEST# 

PREFER  HI  j 

PREFER  LI 

CHI-SQR. 

STAT. 

SIG.  OF  1 
CHI-SQR  1 
STAT.  1 

# 

(%) 

KP 

(%) 

' 

IKES 

HSI 

&29 

■a 

68%  1 

HK9 

S.01 

0.025 

1  3  ; 

27 

68%  | 

33% 

0.022 

(■■1 

BP 

Mill 

HU 

77% 

IlSSiil! 

bbwbi 

IBM 

■a 

5.91 

0.015  II 

1  r 

6 

50% 

6 

50% 

0 

1  | 

1  m 

HB 

PBEI! 

— 1 

iffgj 

— aa 

■ssi 

■S3 

If Ml 

i 

|  0.015  | 

Question  1:  Presentation  of  Case  Information 

This  question  asks  adjudicators  which  case  treatment  writing  style  they  preferred. 
The  results,  statistically  significant  at  the  1%  level,  show  that  by  a  29  to  11  margin 
adjudicators  prefer  the  high-impact  style  to  the  low-impac*  one. 

These  results  confirm  the  findings  from  Part  II  of  the  survey.  Specifically, 
comparing  Part  II  high-impact  and  low-impact  results  of  adjudicators’  impressions  of  the 
way  derogatory  information  is  presented  reveals  that  the  high-impact  style  seems  to 

•  make  it  easier  to  apply  the  adjudication  criteria 

•  increase  the  adjudicators’  confidence  in  the  quality  of  their  adjudication  decision 

•  improve  field  at  nt  awareness  of  adjudicators’  daily  reading  requirements 

•  make  it  easier  to  find  and  remember  derogatory  information 
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Question  2:  Ease  of  Reading 


This  question  asked  the  adjudicators  which  case  style  was  easier  to  read.  The 
results,  statistically  significant  at  the  5%  level,  snow  that  by  a  27  to  13  margin 
adjudicators  found  it  easier  to  read  cases  in  the  high-impact  rather  than  the  low-impact 
style. 


These  results  directly  verify  findings  of  one  Part  II  question  and  indirectly 
corroborate  the  findings  from  two  others.  The  directly  applicable  questions  asked  the 
adjudicators  how  easy  it  was  to  read  the  derogatory  information,  while  the  indirect 
questions  determined  to  what  extent  reading  the  cases  caused  fatigue  and  eye  strain. 

The  high-impact  style  made  it  easier  to  read  derogatory  information  and  reduced  fatigue 
and  eye  strain. 


Question  3;  Ease  of  Remembering  Case  Information 

This  question  asked  adjudicators  if  information  was  easier  to  remember  with  fl  ; 
case  treatments  using  either  the  high-  or  low-impact  style.  The  results,  statistically 
significant  at  the  5%  or  10%  level,  show  that  by  a  27  to  13  margin  adjudicators  more 
easily  remembered  case  information  written  in  the  high-impact  style  than  the  low-impact 
one. 


These  results  confirm  the  finding  from  one  Part  II  question.  This  question  found 
that,  given  the  way  information  was  presented  in  high-  and  low-impact  cases,  adjudicators 
felt  that  it  was  easier  to  remember  the  case  information  in  the  high-impact  style  than  the 
low-impact  one. 


Question  4:  Frequency  of  Rereading  Case  Information 

This  question  asked  adjudicators  which  case  style  had  to  be  read  more  often.  The 
results,  statistically  significant  at  the  1%  level,  show  that  by  a  30  to  9  margin  adjudicators 
had  to  reread  the  case  information  more  often  in  the  low-impact  than  the  high-impact 
cases. 


These  results  corroborate  the  finding  from  one  Part  II  question.  This  question 
determined  that  adjudicators’  felt  they  had  to  reread  sentences  more  often  in  the  low- 
impact  style  than  the  high-impact  one. 


Question  5:  Ease  of  Applying  the  Adjudication  Criteria 

This  question  asked  adjudicators  which  writing  style  made  applying  the  adjudication 
criteria  easier.  The  results,  statistically  significant  at  the  5%  level,  show  that  by  a  27  to 
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12  margin  adjudicators  find  applying  the  adjudication  criteria  easier  in  the  high-impact 
style  than  the  low  one. 

These  results  corroborate  the  finding  from  one  question  in  Part  II.  The  results  to 
that  question  indicate  that  the  adjudicators’  impressions  of  the  way  derogatory 
information  is  presented  in  the  high-impact  style  seems  to  make  it  easier  to  apply  the 
adjudication  criteria  than  in  the  low-impact  cases. 


Question  6:  Confidence  in  Adjudication  Decision 

This  yes/no  question  directly  asked  adjudicators  if  the  different  writing  styles 
affected  their  confidence  in  their  adjudication  decision.  By  a  31  to  9  margin  the 
adjudicators  felt  the  different  styles  did  not  affect  their  confidence. 

Much  care  must  be  used  when  comparing  these  results  with  the  findings  from 
related  Part  II  questions.  The  key  obstacle  to  effective  comparison  lies  in  the  fact  that 
this  Part  IV  question  directly  tested  adjudicators’  confidence  in  their  adjudication 
decision,  while  the  Part  II  questions  determined  only  indirectly  their  degree  of  confidence. 
Part  II  determined  adjudicators’  relative  confidence  in 

•  the  quality  of  the  field  agent’s  investigation 

•  and  the  soundness  of  their  adjudication  decision 

These  results  indirectly  indicate  that  the  high-impact  style  increased  adjudicators’ 
confidence  in  their  decisions  compared  to  the  low-impact  style,  but  the  position  of  the 
mean  scores  shows  that  they  felt  relatively  confident  using  either  style.  Combining  the 
Part  II  and  IV  results,  we  see  that  indirect  testing  shows  that  the  high-impact  style  affects 
adjudicators’  confidence  in  both  the  field  agents’  investigation  and  the  soundness  of  their 
adjudication  decision,  though  direct  testing  does  not. 

However,  we  must  not  ignore  the  fact  that  24%  of  the  adjudicators  did  indicate 
that  the  different  case  writing  styles  did  affect  the  confidence  in  their  adjudication 
decision.  The  other  76%  of  adjudicators  may  be  reluctant  to  say  directly  that  any  case 
style  would  affect  their  confidence  because  this  would  reflect  poorly  on  their  adjudicative 
ability. 


Question  8  and  9:  Perception  of  the  Field  Agent’s  Competence 

Question  #8  was  a  yes/no  question  that  asked  adjudicators  if  the  different  writing 
styles  affected  their  perception  of  the  field  agents’  competence.  Question  #9  asked  only 
those  who  responded  "yes"  if  the  writer  of  the  high-impact  or  the  low-impact  style  was 
more  competent. 
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Question  #8  results  show  that  half  of  the  adjudicators  (20)  indicated  writing  style 
affected  their  perception  of  field  agent  competence.  The  fact  that  half  of  the 
adjudicators  said  that  style  affected  their  perception  is  fairly  startling  since  it  indicates 
that  style  alone  had  a  major  impact  on  perception  of  competence. 

The  competence  of  an  individual  is,  however,  ambiguous,  and  depends  on  the 
definition  used.  It  may  be  a  combination  of  all,  or  part  of,  the  descriptive  adjectives  used 
in  Part  III  of  the  survey.  This  study  interprets  competence  as  the  combination  of  the 
descriptive  adjectives  from  Part  III  that  tested  within  the  significant  level. 

With  this  in  mind,  a  comparison  of  the  Part  IV  and  III  results  reveals  a  pattern 
that  supports  the  finding  that  the  high-impact  style  affects  adjudicators’  perception  of 
field  agents’  competence.  Specifically,  Part  III  shows  that  adjudicators  perceived  field 
agents  who  wrote  the  high-impact  cases  to  be  more  organized,  precise,  careful,  clear- 
thinking,  and  efficient  than  the  writers  of  the  low-impact  cases.  They  also  felt  that  high- 
impact  case  writers  possessed  better  judgment.  The  combination  of  these  statistically 
significant  results  points  out  that  writing  style  does  affect  adjudicators’  perception  of  field 
agents’  competence. 

Furthermore,  the  results  to  Question  #9  show  that  by  a  margin  of  17  to  4  the 
adjudicators  saying  "yes"  to  Question  #8  feel  that  the  writers  of  the  high-impact  cases  are 
more  competent  than  the  writers  who  use  the  low-impact  style.  This  result  also  indicates 
that  the  adjudicators  responding  "yes"  feel  that  high-impact  style  reflects  a  more 
competent  writer  than  the  currently  used  low-impact  style. 


Question  10:  Writing  Style  Preference 

This  question  asked  adjudicators  in  which  style  would  they  prefer  all  ROIs  be 
written.  The  results,  statistically  significant  at  the  5%  level,  show  that  by  a  margin  of  27 
to  12  adjudicators  prefer  the  high-impact  style  cases  over  the  low-impact  style  ones. 

This  result  verifies  the  findings  of  both  Questionnaire  Part  II  and  III.  Those  survey 
results  show  that  there  is  a  statistically  significant  difference  between  the  high-  and  low- 
impact  style  in  the  adjudicators’  impressions  of  case  information  and  their  perceptions  of 
field  agents  writing  the  cases.  Analyzed  together,  the  result  of  Part  II,  III  and  IV  clearly 
point  out  that  the  high-impact  style  is  superior  to  the  present  low-impact  style. 


Analysis  of  the  Separate  Adjudication  Site  Results 

This  section  presents  and  analyzes  the  Part  IV  results  for  each  separate 
adjudication  site.  This  subsection  will  present  qualitative  discussion  of  the  results  from 
the  three  sites:  Army,  Navy  and  DISCR. 
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Presentation  and  Analysis  of  the  Army  Adjudication  Site  Results 

The  Army  site  showed  the  fewest  number  of  statistically  significant  results  to  Part 
IV.  Those  results  are  presented  in  Table  16.  Question  #9,  which  tested  the  positive 
responses  to  question  #8  (the  competence  of  field  agents),  was  significant  at  the  5% 
level,  while  Question  #3,  which  determined  which  case  writing  style  produced  easier-to- 
remember  information,  was  significant  at  the  10%  level.  Although  the  remainder  of  the 
results  are  not  significant,  they  all  favor  the  high-impact  style  documents.  Question  #6 
results  indicate  four  yes  and  eight  no  responses,  while  question  #8  shows  five  yes  and 
seven  no  responses. 

The  only  major  significant  result  showed  by  a  9  to  3  margin  that  adjudicators  felt 
information  presented  in  the  high-impact  style  was  easier  to  remember  than  that 
presented  in  the  low-impact  style.  Also,  the  five  yes  responses  to  question  #8  show  that 
the  respondents  believe  that  the  high-impact  writers  were  more  competent  than  the  low- 
impact  ones. 


TABLE  16 

Summary  of  Questionnaire  Part  IV  Results 
Analysis  of  Army  Site  (n  =  12)  Using  Both  Cases 
Adjudicator’s  Preference  for  High  or  Low-impact  ROI 


QUEST# 

PREFER  HI 

PREFER  LI 

CH1-SQR. 

STAT. 

SIG.  OF 

CHI-SQR. 

STAT. 

# 

(%) 

# 

(%) 

1 

7 

58% 

5 

42% 

0.343 

0.558 

2 

7 

58% 

5 

42% 

0.343 

0.558 

3 

9 

75% 

3 

25% 

3.09 

0.079 

4 

3 

27% 

8 

73% 

2.4 

0.122 

5 

8 

73% 

3 

27% 

2.4 

0.122 

7 

1 

25% 

3 

75% 

NC 

NC 

9 

111 

100% 

^ 1| 

0% 

5 

0.025 

10 

mm 

64% 

■D 

36% 

0.782 

0.376 
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Presentation  and  Analysis  of  the  Naw  Adjudication  Site  Results 


The  Navy  site  showed  the  most  statistically  significant  responses  to  Questionnaire 
Part  IV.  Those  results  are  presented  in  Table  17.  They  show  that  three  questions  (#’s 
1,  2,  and  10)  were  significant  at  the  5%  confidence  level,  and  question  #2  was  significant 
at  the  10%  level.  The  results  also  indicate  that  question  #6  had  only  negative  responses, 
while  question  #8  had  four  positive  and  eight  negative  responses. 

The  major  significant  result  showed  that  the  adjudicators 

•  preferred  the  presentation  in  the  high-impact  style  cases  (#1) 

•  felt  that  the  information  presented  in  the  high-impact  style  was  easier  to  read  (#2) 

•  had  to  reread  information  more  often  in  the  low-impact  style  cases  (#4) 

•  preferred  that  all  cases  be  written  in  the  high-impact  style  (#10) 

Although  the  remainder  of  the  results  are  not  significant,  they  indicate  that  the 
adjudicators  prefer  the  high-impact  documents. 


TABLE  17 

Summary  of  Questionnaire  Part  IV  Results 
Analysis  of  Navy  Site  (n  =  12)  Using  Both  Cases 
Adjudicator’s  Preference  for  High  or  Low-impact  ROI 


QUEST  # 

PREFER  HI 

PREFER  LI 

CHI-SQR. 

STAT. 

SIG.  OF 

CHI-SQR. 

STAT. 

# 

(%) 

# 

(%) 

1 

10 

83% 

2 

17% 

6 ; 

0.014 

2 

9 

75% 

3 

25% 

3.09 

0.079 

3 

8 

67% 

4 

33% 

1.5 

0.221 

4 

1 

8% 

n 

92% 

SS 

0.003 

5 

8 

67% 

4 

33% 

1.5 

0.221 

7 

0 

0% 

0 

0% 

NC 

NC 

9 

4 

80% 

1 

20% 

2.22 

0.136 

10 

IIS 

75% 

3 

■HSU 

0.046 

Presentation  and  Analysis  of  the  DISCR  Adjudication  Site  Results 


The  DISCR  site  showed  the  second  most  statistically  significant  results  to 
Questionnaire  Part  IV.  Those  results  are  presented  in  Table  18.  Although  none  of  the 
responses  were  significant  within  the  5%  level,  and  five  questions  (#’s  1,  2,  5,  9,  and  10  ) 
were  significant  within  10%  level.  The  results  also  indicate  that  question  #6  had  four 
positive  and  eight  negative  responses,  while  question  #8  had  nine  positive  and  three 
negative  responses.  The  statistically  significant  results  showed  that  adjudicators 

•  preferred  the  presentation  in  the  high-impact  style  cases  (#1) 

•  felt  that  the  information  presented  in  the  high-impact  style  was  easier  to  read  (#2) 

•  found  it  easier  to  apply  the  adjudication  criteria  to  information  in  the  high-impact 
style  cases  (#5) 

•  preferred  that  all  cases  be  written  in  the  high-impact  style  (#10) 

Also  the  respondents  who  answered  yes  to  question  #8  indicated  in  question  #9  that 
they  believe  the  high-impact  writers  were  more  competent  than  the  low-impact  ones. 


TABLE  18 


Summary  of  Questionnaire  Part  IV  Results 
Analysis  of  DISCR  Site  (n  =  12)  Using  Both  Cases 
Adjudicator’s  Preference  for  High  or  Low-impact  ROI 


QUEST# 

PREFER  HI  | 

PREFER  LI 

CHI-SQR.  1  SIG.  OF 

STAT.  1  CHI-SQR. 

I  STAT. 

mm 

(%) 

mm 

(%) 

1 

9 

75% 

3 

25% 

3.09 

0.079 

2 

9 

75% 

3 

25% 

3.09 

0.079 

3 

8 

67% 

4 

33% 

1.33 

0.248 

4 

4 

33% 

8 

67% 

1.33 

0.248 

5 

9 

75% 

3 

25% 

3.09 

0.079 

7 

2 

40% 

3 

60% 

0.139 

0.709 

9 

7 

78% 

2 

22% 

2.72 

0.1 

10 

9 

75% 

3 

25% 

3.09 

0.079 
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Although  the  remainder  of  the  results  are  not  statistically  significant,  they  indicate  that 
adjudicators  prefer  the  high-impact  documents. 


Analysis  of  the  Male  Adjudicator  Results 

As  in  the  previous  two  chapters,  the  analysis  of  the  several  demographic 
breakdowns  produced  no  significant  results  except  for  the  male  adjudicator  population. 
Those  results  are  presented  in  Table  19.  Seven  preference  questions  (#’s  1,  2,  3,  4,  5,  9, 
and  10)  were  significant  at  the  5%  level,  and  none  was  significant  within  10%  level.  The 
results  also  indicate  that  question  #6  had  one  positive  and  eight  negative  responses, 
while  question  #8  had  five  positive  and  four  negative  responses.  The  major  significant 
result  showed  that  the  male  adjudicators 

•  preferred  the  presentation  in  the  high-impact  style  cases  (#1) 

•  felt  that  the  information  presented  in  the  high-impact  style  was  easier  to  read  and 
remember  (#’s  2  and  3) 

•  had  to  reread  information  more  often  in  the  low-impact  style  cases  (#4) 


TABLE  19 


Summary  of  Questionnaire  Part  IV  Results 
Analysis  of  Male  Adjudicators  (n  =  9)  Using  Both  Cases 
Adjudicator’s  Preference  for  High  or  Low-impact  ROI 


QUEST  # 

PREFER  HI 

PREFER  LI 

CHI-SQR. 

STAT. 

SIG.  OF  | 
CHI-SQR.  1 
STAT.  I 

# 

(%) 

# 

(%) 

I 

8 

89% 

1 

11% 

5.76 

~  0.016 

2 

8 

m% 

1 

11% 

5.76 

0.016 

89% 

1 

11% 

5.76 

0.016 

4 

i 

11% 

8 

89% 

5.76 

0.016 

5 

8 

89% 

1 

11% 

;  5.76 

0.016 

7 

0 

0% 

2 

100% 

2 

0.157 

9 

'  5 

100% 

0 

0% 

•5 

0.025 

10 

8 

89% 

■  V::T- 

11% 

5.76 

0.016 
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•  found  it  easier  to  apply  the  adjudication  criteria  to  information  in  the  high-impact 
style  cases  (#5) 

•  preferred  that  all  cases  be  written  in  the  high-impact  style  (#10) 

Also  the  respondents  who  answered  yes  to  question  #8  indicated  in  the  question  #9 
results  that  they  believe  that  the  high-impact  writers  were  more  competent  than  the  low- 
impact  ones. 

These  results  confirm  the  findings  in  Part  II  and  III.  The  male  adjudicators  are  the 
mavericks  in  the  adjudication  population.  They  seem  to  believe  more  strongly  than  their 
female  counterparts  that  the  high-impact  style  cases  are  a  great  improvement  over  the 
current  low-impact  ones. 


Chapter  Conclusion 

The  overall  results  of  Part  IV  point  out  that  there  is  statistically  significant 
evidence  to  show  that  adjudicators  prefer  cases  be  written  in  the  high-impact  rather  than 
the  low-impact  style.  Specifically,  the  analysis  indicates  the  adjudicators  prefer  the  high- 
impact  over  the  low-impact  style  relative  to  the  following  factors: 

•  presentation  of  information 

•  ease  of  reading,  remembering,  and  applying  the  adjudication  criteria 

•  amount  of  rereading 

•  overall  writing  style 

When  the  Part  IV  analysis  was  broken  down  by  adjudication  site,  the  statistical 
significance  was  not  nearly  as  great  as  the  results  from  the  entire  population.  This  result 
could  be  attributed  to  the  different  sample  sizes  used  in  the  analysis  of  both  cases  and 
each  site  (i.e.  40  versus  12  adjudicators).  The  greatest  statistical  differences  were  found 
in  the  results  from  the  Navy  adjudicators,  with  DISCR  coming  second,  and  Army  third. 

As  in  the  previous  two  chapters,  the  most  surprising  result  was  found  in  the 
analysis  of  the  demographic  breakdown  of  adjudicators.  The  survey  responses  from  the 
male  adjudicators  produced  statistically  significant  results. 
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Conclusions  and  Recommendations 


Conclusions 


Results  from  Parts  II,  III  and  IV  on  the  high-  and  low-impact  treatment  of  the 
Czamek  and  Rokitka  cases  shows  that  adjudicators  prefer  the  high-impact  style  over  the 
currently  used  low-impact  one.  The  data  indicate  the  adjudicators  prefer  the  high-impact 
style  for  its 

•  presentation  of  information 

•  ease  of  reading,  remembering,  and  applying  the  adjudication  criteria 

•  limited  amount  of  rereading 

•  overall  writing  style 

There  was,  however,  no  statistical  evidence  to  show  that  the  high-impact  case 
decreases  adjudication  time. 

In  the  area  of  the  adjudicators’  impression  of  the  case  information,  the  analysis 
does  produce  statistically  significant  evidence  to  indicate  that  adjudicators  feel  that  the 
high-impact  style  is  an  improvement  over  the  low-impact  style.  Specifically,  they  believe 
that  the  high-impact  style 

•  provides  better  coverage  of  issues 

•  makes  it  easier  to  apply  adjudication  criteria  to  the  case  information 

•  makes  the  field  agent  who  wrote  the  case  seem  more  aware  of  1)  how  to  make  the 
case  easier  to  read;  2)  their  informational  needs;  3)  the  extensive  amount  of  their 
daily  reading 

•  makes  the  field  agent  seem  like  a  better  writer 

•  makes  the  derogatory  information  in  the  narrative  report  easier  to  find,  read, 
remember  and  adjudicate 

•  makes  following  the  chronological  sequence  of  events  in  the  case  easier  to  follow 

•  causes  less  fatigue  while  reading 


65 


To  a  lesser  degree  adjudicators  believe  that  the  high-impact  style  makes  them 
more  confident  in  the  soundness  and  quality  of  their  adjudication  decision,  causes  them 
to  reread  sentences  less  often,  and  produces  less  eye  stain. 

The  analysis  of  the  adjudicators’  perception  of  the  field  agents  provided  statistically 
significant  evidence  to  show  that  the  field  agents  who  wrote  high-impact  style  cases  are 
perceived  differently  than  those  who  composed  low-impact  ones.  Specifically, 
adjudicators  feel  that  the  agents  writing  in  a  high-impact  style  possess  better  judgment, 
are  better  organized,  and  are  more  precise,  careful,  clear  thinking,  and  efficient  than 
writers  of  low-impact  cases.  These  perceived  attributes  may  increase  field  agent 
credibility,  which  could  improve  the  efficiency  of  the  entire  adjudication  process. 

Both  Part  II  and  III  results  indicate  that  there  were  greater  statistically  significant 
differences  between  high-  and  low-impact  case  treatments  of  the  Rokitka  cases  than  the 
Czamek  cases.  This  result  is  contrary  to  expected  results  because  the  Czamek  case 
content  is  more  complicated  and  longer  than  Rokitka  case  content.  One  would  expect 
that  the  longer  and  more  complicated  a  document’s  content,  the  more  helpful  the  high- 
impact  style  would  be  in  reading,  finding,  and  processing  information. 

When  the  analysis  was  divided  by  adjudication  site,  the  statistical  significance  was 
not  nearly  as  great  as  the  result  from  the  entire  population.  However,  this  could  be 
attributed  to  the  smaller  sample  sizes  used.  Overall,  the  greatest  number  of  statistical 
differences  were  found  from  the  analysis  of  the  Navy  adjudication  site  data.  The  Army 
adjudication  site  data  produced  the  second  most  statistically  significant  result,  while 
DISCR  data  came  in  a  distant  third. 

The  most  surprising  result  was  found  when  analyzing  the  demographic  breakdown 
of  adjudicators.  The  male  adjudicators  found  that  the  high-impact  style  decreased  the 
time  required  to  adjudicate  a  case.  They  also  felt  that  the  high-impact  cases  were  a 
much  greater  improvement  over  the  current  low-impact  cases.  Finally,  their  perception 
of  the  field  agents  who  wrote  the  high-impact  cases  was  extremely  different  than  the  low- 
impact  authors.  These  results  may  be  due  to  the  fact  that  males  are  a  minority  and 
could  be  considered  outsiders.  Outsiders  may  be  more  willing  to  change  the  status  quo 
of  an  organization  than  insiders. 


Recommendations 


The  overall  results  of  Questionnaire  Parts  II,  III,  and  IV  clearly  shows  that  the 
adjudicators  perceive  the  high-impact  style  cases  to  be  superior  to  the  current  low-impact 
style.  Specifically,  most  adjudicators  prefer  that  all  ROIs  be  written  in  a  high-impact 
style. 


With  this  in  mind,  DIS  should  heed  their  customers’  desires  and  train  their  field 
agents  to  write  ROIs  in  a  high-impact  style.  Although  the  high-impact  style  does  not  at  a 
statistically  significant  level  improve  the  decision-making  ability  or  decrease  case 
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assessment  time,  we  still  believe  that  DIS  should  adopt  the  high-impact  style  in  the  spirit 
of  improved  customer  support. 

If  implemented,  the  high-impact  ROIs  could  increase  the  efficiency  of  the 
adjudication  process.  The  adjudicators  will  be  seeing  cases  written  in  a  way  they  perceive 
as  more  efficient.  This  perceived  efficiency  may  increase  the  field  agents’  credibility  in 
the  eyes  of  adjudicators  and  thus  produce  greater  adjudicator  confidence  in  the  ROI  case 
information.  This  increased  credibility  and  confidence  in  the  case  information  could 
reduce  requests  for  additional  information  concerning  the  case  subject.  This  reduction  in 
requests  for  additional  information  will  thus  decrease  overall  case  processing  time. 
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APPENDIX  A 


PART  II 


INSTRUCTIONS 

This  form  is  designed  to  capture  your  impressions  of  the  case  information  you’ve  read 
and  the  adjudication  process  you've  undergone.  Please  read  each  of  the  following 
statements  carefully,  and  then  place  an  *XW- next  to  the  place  on  the  scale  that  best 
describes  your  response  to  each  statement. 

For  example,  if  you  were  responding  to  the  statement 

It  was _ for  me  to  find  important  information  in  the  case. 


□  □  □  □ 

12  3  4 

Easy 

and  you  felt  that  to  End  information  it  was 


□ 

s 

Difficult 


easy: 

check 

somewhat  easy: 

check 

neither  easy 

nor  difficult: 

check 

somewhat  difficult: 

check 

difficult: 

check 

I 


STATEMENTS 


1.  This  case  provided _ coverage  of  issues  to  enable  me  to  make  a  high  quality 

adjudication  decision. 

□  anna 

1  2  3  4  5 

Good  Poor 
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2. 

The  way  field  agents  presented  derogatory  information  in  the  case  made  it  .  , 

to  apply  adjudication  criteria  to  that  information. 

□  □ 

□ 

□ 

□ 

1  2 

3 

4  S 

1 

Easy 

Difficult 

3. 

The  way  derogatory  information  was 

presented  made  me  ,  _  in  the  quality  of 

the  field  agents’  investigation. 

□ 

□ 

□ 

□ 

□ 

1  2 

3 

4  S 

Confident 

Unconfident 

4. 

After  readme  the  case,  I  am 

in  the  soundness  of  my  adjudication  decision. 

□ 

□ 

□ 

□ 

□ 

1  2 

3 

4  5 

Confident 

Unconfident 

5. 

The  field  a  cents  seemed  to  be 

of  the  information  I  needed  to  make  a  high 

quality  adjudication  decision. 

□ 

□ 

□ 

□ 

□ 

1  2 

3 

4  5 

Aware 

Unaware 

6. 

The  field  atrents  who  wrote  this  case  seemed  to  be  of  how  to  make  the 

case  easy  for  me  to  read. 

□ 

□ 

□ 

□ 

□ 

1  2 

3 

4  5 

Aware 

Unaware 

7  4 

7.  Based  on  the  way  field  agents  presented  information  in  the  case,  they  seemed  to  be 
_ _  of  the  large  amounts  of  information  I  have  to  read  daily. 


□  □  □  □  □ 


Aware 


5 

Unaware 


8.  I  believe  the  Geld  agents  who  wTOte  the  narrative  reports  for  this  case  are 
writers. 

□  □  □  □  □ 


9.  I _ that  the  field  agent  who  wrote  the  case's  subject  interview  section  should 

receive  training  to  make  his/her  writing  more  readable. 


□  □  □  □  □ 


Agree 


Disagree 


10.  I  was  able  to  read  this  case 


than  similar  cases. 


D  □  □  □  □ 


Faster 


Slower 
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11.  The  way  field  agents  presented  information  in  their  narrative  repons  made  it 
_ to  find  derogatory  information. 

□  □  □  D  □ 


Difficult 


12.  Derogatory  information  written  by  the  field  agents  was _ 

□  □  □  □  □ 

1  2  3  4  5 

Easy  Difficult 


to  read. 


13.  I  thought  the  subject  interview  section  of  the  case  was _ 

□  □  □  □  □ 

1  2  3  4  5 

Easy  Difficult 


to  read. 


14.  The  way  the  field  agent  presented  derogatory  information  made  it 
remember  that  information. 

□  □  □  □  □ 


Difficult 


15.  I  found  the  case  to  be 


to  adjudicate. 


□  □  □  □  □ 


Difficult 
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16.  It  was _ for  me  to  locate  information  essential  to  my  efficient  adjudication 

of  the  case. 

□  □  □  □  □ 

1  2  3  4  5 

Easy  DifGcult 


17.  To  understand  what  the  Geld  agenr  was  trying  to  convey,  I _ had  to  reread 

a  section  or  sections  of  the  case. 

□  □  □  □  □ 

1  2  3  4  5 

Often  Never 


18.  I _ had  to  reread  sentences  to  understand  the  exact  idea  Geld  agents  were 

trying  to  convey. 


c 

!  □ 

□ 

□ 

□ 

1 

2 

3 

4 

5 

Often 

Never 

19.  I  found  it _ 

to 

follow  the 

chronological 

□ 

sequence  of  events  in  the  case. 

□ 

□ 

□ 

□ 

i 

2 

3 

4 

5 

Easy 

DifGcult 

20.  To  help  me  understand  the  logical  connection  between  ideas  in  the  case,  I 
_  had  to  either  underline  parts  of  the  case  or  make  notes. 


□  □  □ 

1  2  3 

Often 


□  □ 

4  5 

Never 
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21.  1 _ encountered  words  that  were  difficult  to  understand  while  reading  the 


Held  agents'  narratives. 

□ 

□ 

□ 

□ 

□ 

i 

2 

3 

4 

s 

Often 

Never 

Reading  this  case  tired  me  out. 

□ 

□ 

□ 

□ 

□ 

1 

2 

3 

4 

5 

Agree 

Disagree 

Reading  this  case  strained  my  eyes. 

□ 

□ 

□ 

□ 

□ 

t 

2 

3 

4 

5 

Agree 

Disagree 
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APPENDIX  B 


PART  HI 


INSTRUCTIONS 

Please  indicate  your  perceptions  or  feelings  toward  the  Geld  agents  who  wrote  the  ROI 
you've  just  read  by  placing  an  "X"  near  the  item  that  best  describes  that  perception. 


For  example,  if  after  reading  the  RQL  you  feel  the  Geld  agents 

Thorough: 

check  #  1 

Somewhat  thorough: 

check  #  2 

Neither  thorough 
nor  careless: 

check  #  3 

Somewhat  careless: 

check  #  4 

Careless: 

check  #  3 

Use  this  guideline  to  respond  to  the  following  items. 

PERCEPTIONS  OF  FIELD  AGENTS 

i.  1 _ 1 _ 1 _ 1 _ J _ 1 

1  2  3 

4  5 

Thorough 

Careless 

2-  1  1  1 

1  1  1 

1  2  3 

4  5 

Competent 

Incompetent 

3.  1  !  I 

1  1  ! 

1  2  3 

4  5 

Objective 

Biased 
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PERCEPTIONS  OF  FIELD  AGENTS  (Continued) 


Perceptive 


Unperceptive 


Organized 


Disorganized 


1  2  3 

Precise 


Imprecise 


Careful 


4  5 

Careless 


Clear  thinking 


Fuzzy  thinking 
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PERCEPTIONS  OF  FIELD  AGENTS  (Continued) 


10. 

i 

Efficient 

2 

3 

4 

s 

Inefficient 

11. 

L  ....  1 

! 

1  1 

1 

Confident 

2 

3 

4 

S 

Unconfident 

12 

1  1 

I 

1  1 

1 

Reliable 

2 

3 

4 

5 

Unreliable 

13. 

1  1 

I 

! 

1  1 

1 

Experienced 

2 

3 

4 

s 

Inexperienced 

14. 

1 _ L 

_ 1 

1  1 

1  2 
Sound  judgment 

3 

4 

5 

Poor  judgment 

15 

1  1 

i 

!  1 

i 

Veteran 

2 

3 

4 

5 

Rookie 
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APPENDIX  C 


PART  IV 

Information  presented  in  both  cases  was  written  and  formatted  in  different  styles.  Please 
provide  us  with  your  impressions  or  feelings  about  these  different  written  styles  by  checking 
the  appropriate  responses  to  the  statements  listed  below. 

1.  I  prefened  the  way  information  was  presented  in 

Case  I  _ 

Case  II _ 

2.  I  felt  the  case  that  was  easier  to  read  was 

Case  I  _ 

Case  II _ 

3.  Information  was  easier  to  remember  in 

Case  I  _ 

Case  II _ 

4.  I  had  to  reread  information  more  often  in 

Case  I  _ 

Case  II _ 

5.  I  found  it  easier  to  apply  adjudication  criteria  to  information  in 

Case  I  _ 

Case  II _ 
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6.  1  felt  the  two  cases’  different  writing  styles  affected  my  confidence  in  my  adjudication 
decision. 

Yes  _ 

No  _ 

7.  If  you  answered  "yes''  to  question  6,  did  you  feel  more  confident  of  your  decision  when 
you  adjudicated  Case  1  or  Case  II? 

Case  I  _ 

Case  II _ 

8.  I  felt  the  cases’  different  written  communication  styles  affected  my  perception  of  the 
competence  of  the  field  agents. 

Yes  _ 

No  _ 

9.  If  you  answered  "yes"  to  question  8,  did  you  feel  that  the  field  agents  who  wrote  Case 
I  or  Case  II  were  more  competent? 

Case  I  _ 

Case  II _ 

10.  I  would  prefer  that  all  ROIs  be  written  in  the  style  of 


